[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781947562949448-16558-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:26:03 dist-test-slave-706q krb5kdc[16565](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:26:03 dist-test-slave-706q krb5kdc[16565](info): set up 2 sockets
Jun 20 09:26:03 dist-test-slave-706q krb5kdc[16565](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260620 09:26:04.009986 16558 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781947562949448-16558-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781947562949448-16558-0/krb5kdc/test-user.keytab.
Jun 20 09:26:04 dist-test-slave-706q krb5kdc[16565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947564, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:26:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:26:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.43.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.43.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.43.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.190.keytab.
Entry for principal kudu/127.16.43.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.190.keytab.
I20260620 09:26:04.134299 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781947562949448-16558-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781947562949448-16558-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781947562949448-16558-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.43.190:44389
--webserver_interface=127.16.43.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.190.keytab
--principal=kudu/127.16.43.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.43.148:37453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.43.190:44389
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781947562949448-16558-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781947562949448-16558-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781947562949448-16558-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:26:04.216037 16578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:26:04.216269 16578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:26:04.216290 16578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:26:04.217374 16578 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:26:04.217410 16578 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:26:04.217424 16578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:26:04.217437 16578 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:26:04.217449 16578 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:26:04.218850 16578 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:37453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781947562949448-16558-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781947562949448-16558-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.43.190:44389
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.190.keytab
--principal=kudu/127.16.43.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.43.190:44389
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.43.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16578
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781947562949448-16558-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:26:04.219051 16578 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:26:04.219231 16578 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:26:04.224251 16583 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:26:04.224251 16584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:26:04.224367 16586 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:26:04.224392 16578 server_base.cc:1034] running on GCE node
I20260620 09:26:04.224841 16578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:26:04.225111 16578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:26:04.226261 16578 hybrid_clock.cc:648] HybridClock initialized: now 1781947564226247 us; error 28 us; skew 500 ppm
Jun 20 09:26:04 dist-test-slave-706q krb5kdc[16565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947564, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:26:04.229027 16578 init.cc:377] Logged in from keytab as kudu/127.16.43.190@KRBTEST.COM (short username kudu)
I20260620 09:26:04.229487 16578 webserver.cc:460] Webserver started at http://127.16.43.190:33853/ using document root <none> and password file <none>
I20260620 09:26:04.229748 16578 fs_manager.cc:359] Metadata directory not provided
I20260620 09:26:04.229820 16578 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:26:04.229975 16578 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:26:04.231078 16578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781947562949448-16558-0/minicluster-data/master-0/data/instance:
uuid: "75c6dad15db74889b23425c40be5e4df"
format_stamp: "Formatted at 2026-06-20 09:26:04 on dist-test-slave-706q"
server_key: "fa34ebae0e4bd449595d7c3e49070090"
server_key_iv: "697cdc8e8cfa87fde9aee34bc554b9ba"
server_key_version: "encryptionkey@0"
I20260620 09:26:04.231458 16578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781947562949448-16558-0/minicluster-data/master-0/wal/instance:
uuid: "75c6dad15db74889b23425c40be5e4df"
format_stamp: "Formatted at 2026-06-20 09:26:04 on dist-test-slave-706q"
server_key: "fa34ebae0e4bd449595d7c3e49070090"
server_key_iv: "697cdc8e8cfa87fde9aee34bc554b9ba"
server_key_version: "encryptionkey@0"
I20260620 09:26:04.232833 16578 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 09:26:04.233572 16593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:04.233765 16578 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 09:26:04.233866 16578 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781947562949448-16558-0/minicluster-data/master-0/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781947562949448-16558-0/minicluster-data/master-0/wal
uuid: "75c6dad15db74889b23425c40be5e4df"
format_stamp: "Formatted at 2026-06-20 09:26:04 on dist-test-slave-706q"
server_key: "fa34ebae0e4bd449595d7c3e49070090"
server_key_iv: "697cdc8e8cfa87fde9aee34bc554b9ba"
server_key_version: "encryptionkey@0"
I20260620 09:26:04.233947 16578 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781947562949448-16558-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781947562949448-16558-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781947562949448-16558-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:26:04.252748 16578 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:26:04.254844 16578 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:26:04.255015 16578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:26:04.259707 16578 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.190:44389
I20260620 09:26:04.259744 16645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.190:44389 every 8 connection(s)
I20260620 09:26:04.260106 16578 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
I20260620 09:26:04.261049 16646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:26:04.263432 16646 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75c6dad15db74889b23425c40be5e4df: Bootstrap starting.
I20260620 09:26:04.264035 16646 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 75c6dad15db74889b23425c40be5e4df: Neither blocks nor log segments found. Creating new log.
I20260620 09:26:04.264350 16646 log.cc:824] T 00000000000000000000000000000000 P 75c6dad15db74889b23425c40be5e4df: Log is configured to *not* fsync() on all Append() calls
I20260620 09:26:04.265054 16646 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75c6dad15db74889b23425c40be5e4df: No bootstrap required, opened a new log
I20260620 09:26:04.266774 16646 raft_consensus.cc:348] T 00000000000000000000000000000000 P 75c6dad15db74889b23425c40be5e4df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75c6dad15db74889b23425c40be5e4df" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 44389 } }
I20260620 09:26:04.266925 16646 raft_consensus.cc:374] T 00000000000000000000000000000000 P 75c6dad15db74889b23425c40be5e4df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:26:04.266950 16646 raft_consensus.cc:729] T 00000000000000000000000000000000 P 75c6dad15db74889b23425c40be5e4df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75c6dad15db74889b23425c40be5e4df, State: Initialized, Role: FOLLOWER
I20260620 09:26:04.267118 16646 consensus_queue.cc:260] T 00000000000000000000000000000000 P 75c6dad15db74889b23425c40be5e4df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75c6dad15db74889b23425c40be5e4df" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 44389 } }
I20260620 09:26:04.267185 16646 raft_consensus.cc:388] T 00000000000000000000000000000000 P 75c6dad15db74889b23425c40be5e4df [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:26:04.267208 16646 raft_consensus.cc:482] T 00000000000000000000000000000000 P 75c6dad15db74889b23425c40be5e4df [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:26:04.267266 16646 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 75c6dad15db74889b23425c40be5e4df [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:26:04.267905 16646 raft_consensus.cc:504] T 00000000000000000000000000000000 P 75c6dad15db74889b23425c40be5e4df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75c6dad15db74889b23425c40be5e4df" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 44389 } }
I20260620 09:26:04.268023 16646 leader_election.cc:302] T 00000000000000000000000000000000 P 75c6dad15db74889b23425c40be5e4df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 75c6dad15db74889b23425c40be5e4df; no voters:
I20260620 09:26:04.268219 16646 leader_election.cc:288] T 00000000000000000000000000000000 P 75c6dad15db74889b23425c40be5e4df [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:26:04.268266 16649 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 75c6dad15db74889b23425c40be5e4df [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:26:04.268446 16649 raft_consensus.cc:686] T 00000000000000000000000000000000 P 75c6dad15db74889b23425c40be5e4df [term 1 LEADER]: Becoming Leader. State: Replica: 75c6dad15db74889b23425c40be5e4df, State: Running, Role: LEADER
I20260620 09:26:04.268550 16646 sys_catalog.cc:564] T 00000000000000000000000000000000 P 75c6dad15db74889b23425c40be5e4df [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:26:04.268572 16649 consensus_queue.cc:237] T 00000000000000000000000000000000 P 75c6dad15db74889b23425c40be5e4df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75c6dad15db74889b23425c40be5e4df" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 44389 } }
I20260620 09:26:04.269032 16650 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75c6dad15db74889b23425c40be5e4df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "75c6dad15db74889b23425c40be5e4df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75c6dad15db74889b23425c40be5e4df" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 44389 } } }
I20260620 09:26:04.269191 16650 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75c6dad15db74889b23425c40be5e4df [sys.catalog]: This master's current role is: LEADER
I20260620 09:26:04.269454 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 16578
I20260620 09:26:04.269562 16558 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781947562949448-16558-0/minicluster-data/master-0/wal/instance
I20260620 09:26:04.269706 16654 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:26:04.269066 16651 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75c6dad15db74889b23425c40be5e4df [sys.catalog]: SysCatalogTable state changed. Reason: New leader 75c6dad15db74889b23425c40be5e4df. Latest consensus state: current_term: 1 leader_uuid: "75c6dad15db74889b23425c40be5e4df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75c6dad15db74889b23425c40be5e4df" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 44389 } } }
I20260620 09:26:04.269869 16651 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75c6dad15db74889b23425c40be5e4df [sys.catalog]: This master's current role is: LEADER
I20260620 09:26:04.270387 16558 external_mini_cluster.cc:1428] Setting key d01ec1842461fe6373775614632d2aba
I20260620 09:26:04.270426 16654 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:26:04.276036 16654 catalog_manager.cc:1347] Generated new cluster ID: 22b1a6f3301f42eb8408bf9131f1a19e
I20260620 09:26:04.276110 16654 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 09:26:04 dist-test-slave-706q krb5kdc[16565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947564, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.43.190@KRBTEST.COM
I20260620 09:26:04.287947 16668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:04.271416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50634 (local address 127.16.43.190:44389)
0620 09:26:04.271589 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:04.271600 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:04.271816 (+ 216us) server_negotiation.cc:407] Connection header received
0620 09:26:04.273623 (+ 1807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:04.273631 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:04.273687 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:04.273846 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:04.275343 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:04.276956 (+ 1613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:04.278415 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:04.279116 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:04.279231 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:04.283568 (+ 4337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:04.283573 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:04.283573 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:26:04.283585 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:04.285641 (+ 2056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:04.286173 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:04.286176 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:04.286176 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:04.286293 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:04.286584 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:04.286585 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:04.286585 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:04.286888 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:04.287040 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:04.287356 (+ 316us) server_negotiation.cc:299] Negotiation successful
0620 09:26:04.287490 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":78,"threads_started":1}
I20260620 09:26:04.289858 16654 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:26:04.290369 16654 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:26:04.311893 16654 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 75c6dad15db74889b23425c40be5e4df: Generated new TSK 0
I20260620 09:26:04.312177 16654 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.43.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.43.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.43.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.129.keytab.
Entry for principal kudu/127.16.43.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.129.keytab.
I20260620 09:26:04.343551 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781947562949448-16558-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781947562949448-16558-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781947562949448-16558-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781947562949448-16558-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.43.129:0
--local_ip_for_outbound_sockets=127.16.43.129
--webserver_interface=127.16.43.129
--webserver_port=0
--tserver_master_addrs=127.16.43.190:44389
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.129.keytab
--principal=kudu/127.16.43.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.43.148:37453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781947562949448-16558-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781947562949448-16558-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781947562949448-16558-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:26:04.430503 16672 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:26:04.430707 16672 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:26:04.430732 16672 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:26:04.431919 16672 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:26:04.431968 16672 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:26:04.431998 16672 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.43.129
I20260620 09:26:04.433454 16672 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:37453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781947562949448-16558-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781947562949448-16558-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.129.keytab
--principal=kudu/127.16.43.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.43.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781947562949448-16558-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.43.129
--webserver_port=0
--tserver_master_addrs=127.16.43.190:44389
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16672
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.43.129
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781947562949448-16558-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:26:04.433717 16672 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:26:04.433979 16672 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:26:04.439476 16677 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:26:04.439468 16680 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:26:04.439468 16678 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:26:04.439703 16672 server_base.cc:1034] running on GCE node
I20260620 09:26:04.440011 16672 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:26:04.440274 16672 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:26:04.441481 16672 hybrid_clock.cc:648] HybridClock initialized: now 1781947564441449 us; error 72 us; skew 500 ppm
Jun 20 09:26:04 dist-test-slave-706q krb5kdc[16565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947564, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:26:04.444275 16672 init.cc:377] Logged in from keytab as kudu/127.16.43.129@KRBTEST.COM (short username kudu)
I20260620 09:26:04.444785 16672 webserver.cc:460] Webserver started at http://127.16.43.129:39375/ using document root <none> and password file <none>
I20260620 09:26:04.445016 16672 fs_manager.cc:359] Metadata directory not provided
I20260620 09:26:04.445091 16672 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:26:04.445241 16672 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:26:04.446297 16672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781947562949448-16558-0/minicluster-data/ts-0/data/instance:
uuid: "39e6a8f460bf4aa3b6c8ec8a09103478"
format_stamp: "Formatted at 2026-06-20 09:26:04 on dist-test-slave-706q"
server_key: "577ebdea69274d5b302b5ddfd97febc6"
server_key_iv: "17ce68bba23157befad3789b3bc9435d"
server_key_version: "encryptionkey@0"
I20260620 09:26:04.446674 16672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781947562949448-16558-0/minicluster-data/ts-0/wal/instance:
uuid: "39e6a8f460bf4aa3b6c8ec8a09103478"
format_stamp: "Formatted at 2026-06-20 09:26:04 on dist-test-slave-706q"
server_key: "577ebdea69274d5b302b5ddfd97febc6"
server_key_iv: "17ce68bba23157befad3789b3bc9435d"
server_key_version: "encryptionkey@0"
I20260620 09:26:04.448173 16672 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 09:26:04.448918 16687 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:04.449095 16672 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 09:26:04.449195 16672 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781947562949448-16558-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781947562949448-16558-0/minicluster-data/ts-0/wal
uuid: "39e6a8f460bf4aa3b6c8ec8a09103478"
format_stamp: "Formatted at 2026-06-20 09:26:04 on dist-test-slave-706q"
server_key: "577ebdea69274d5b302b5ddfd97febc6"
server_key_iv: "17ce68bba23157befad3789b3bc9435d"
server_key_version: "encryptionkey@0"
I20260620 09:26:04.449273 16672 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781947562949448-16558-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781947562949448-16558-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781947562949448-16558-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:26:04.470055 16672 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:26:04.472630 16672 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:26:04.472822 16672 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:26:04.473142 16672 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:26:04.473470 16672 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:26:04.473526 16672 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:04.473567 16672 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:26:04.473595 16672 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:04.479661 16672 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.129:44079
I20260620 09:26:04.479728 16800 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.129:44079 every 8 connection(s)
I20260620 09:26:04.480098 16672 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781947562949448-16558-0/minicluster-data/ts-0/data/info.pb
Jun 20 09:26:04 dist-test-slave-706q krb5kdc[16565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947564, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.129@KRBTEST.COM for kudu/127.16.43.190@KRBTEST.COM
I20260620 09:26:04.488817 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 16672
I20260620 09:26:04.488933 16558 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781947562949448-16558-0/minicluster-data/ts-0/wal/instance
I20260620 09:26:04.489108 16558 external_mini_cluster.cc:1428] Setting key 7d5497c0430d67711a0177f5f355c1ec
I20260620 09:26:04.490116 16668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:04.480996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.129:53767 (local address 127.16.43.190:44389)
0620 09:26:04.481091 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:04.481093 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:04.481251 (+ 158us) server_negotiation.cc:407] Connection header received
0620 09:26:04.481759 (+ 508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:04.481761 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:04.481774 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:04.481839 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:04.482892 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:04.483475 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:04.484572 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:04.484783 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:04.484853 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:04.487139 (+ 2286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:04.487141 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:04.487141 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:26:04.487148 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:04.488567 (+ 1419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:04.488997 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:04.488998 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:04.488999 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:04.489060 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:04.489356 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:04.489357 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:04.489357 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:04.489486 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:04.489569 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:04.489935 (+ 366us) server_negotiation.cc:299] Negotiation successful
0620 09:26:04.490027 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 09:26:04.490309 16803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:04.480917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:44389 (local address 127.16.43.129:53767)
0620 09:26:04.481121 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:04.481163 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:04.481650 (+ 487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:04.481921 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:04.481924 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:04.482569 (+ 645us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:04.482770 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:04.482775 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:04.483676 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:04.483676 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:04.484474 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:04.484476 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:04.484865 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:04.484866 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:04.484963 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:04.485584 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:26:04.485593 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:26:04.487050 (+ 1457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:26:04.488646 (+ 1596us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:04.488652 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:04.488654 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:04.488869 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:04.489132 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:04.489133 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:04.489134 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:04.489275 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:04.489574 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:26:04.489582 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:26:04.489766 (+ 184us) client_negotiation.cc:769] Sending connection context
0620 09:26:04.489892 (+ 126us) client_negotiation.cc:240] Negotiation successful
0620 09:26:04.490060 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":107,"threads_started":1}
I20260620 09:26:04.491744 16801 heartbeater.cc:347] Connected to a master server at 127.16.43.190:44389
I20260620 09:26:04.491844 16801 heartbeater.cc:464] Registering TS with master...
I20260620 09:26:04.492046 16801 heartbeater.cc:511] Master 127.16.43.190:44389 requested a full tablet report, sending...
I20260620 09:26:04.492686 16610 ts_manager.cc:194] Registered new tserver with Master: 39e6a8f460bf4aa3b6c8ec8a09103478 (127.16.43.129:44079)
I20260620 09:26:04.493346 16610 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.43.129@KRBTEST.COM'} at 127.16.43.129:53767
WARNING: no policy specified for kudu/127.16.43.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.43.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.43.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.130.keytab.
Entry for principal kudu/127.16.43.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.130.keytab.
I20260620 09:26:04.515328 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781947562949448-16558-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781947562949448-16558-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781947562949448-16558-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781947562949448-16558-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.43.130:0
--local_ip_for_outbound_sockets=127.16.43.130
--webserver_interface=127.16.43.130
--webserver_port=0
--tserver_master_addrs=127.16.43.190:44389
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.130.keytab
--principal=kudu/127.16.43.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.43.148:37453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781947562949448-16558-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781947562949448-16558-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781947562949448-16558-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:26:04.597299 16806 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:26:04.597510 16806 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:26:04.597535 16806 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:26:04.598657 16806 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:26:04.598704 16806 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:26:04.598731 16806 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.43.130
I20260620 09:26:04.600204 16806 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:37453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781947562949448-16558-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781947562949448-16558-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.130.keytab
--principal=kudu/127.16.43.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.43.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781947562949448-16558-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.43.130
--webserver_port=0
--tserver_master_addrs=127.16.43.190:44389
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16806
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.43.130
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781947562949448-16558-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:26:04.600394 16806 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:26:04.600577 16806 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:26:04.605669 16811 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:26:04.605818 16814 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:26:04.605679 16812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:26:04.605928 16806 server_base.cc:1034] running on GCE node
I20260620 09:26:04.606194 16806 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:26:04.606400 16806 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:26:04.607615 16806 hybrid_clock.cc:648] HybridClock initialized: now 1781947564607606 us; error 34 us; skew 500 ppm
Jun 20 09:26:04 dist-test-slave-706q krb5kdc[16565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947564, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:26:04.610278 16806 init.cc:377] Logged in from keytab as kudu/127.16.43.130@KRBTEST.COM (short username kudu)
I20260620 09:26:04.610739 16806 webserver.cc:460] Webserver started at http://127.16.43.130:33377/ using document root <none> and password file <none>
I20260620 09:26:04.610981 16806 fs_manager.cc:359] Metadata directory not provided
I20260620 09:26:04.611057 16806 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:26:04.611207 16806 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:26:04.612336 16806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781947562949448-16558-0/minicluster-data/ts-1/data/instance:
uuid: "84fe1c76645d4914b4f0231dc0a7c9a2"
format_stamp: "Formatted at 2026-06-20 09:26:04 on dist-test-slave-706q"
server_key: "a231aac6156b7bdedd34b510a4b5202e"
server_key_iv: "27ae32f88ad61da3f5c33887f5099e78"
server_key_version: "encryptionkey@0"
I20260620 09:26:04.612707 16806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781947562949448-16558-0/minicluster-data/ts-1/wal/instance:
uuid: "84fe1c76645d4914b4f0231dc0a7c9a2"
format_stamp: "Formatted at 2026-06-20 09:26:04 on dist-test-slave-706q"
server_key: "a231aac6156b7bdedd34b510a4b5202e"
server_key_iv: "27ae32f88ad61da3f5c33887f5099e78"
server_key_version: "encryptionkey@0"
I20260620 09:26:04.614089 16806 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 09:26:04.614861 16821 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:04.615124 16806 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:26:04.615226 16806 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781947562949448-16558-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781947562949448-16558-0/minicluster-data/ts-1/wal
uuid: "84fe1c76645d4914b4f0231dc0a7c9a2"
format_stamp: "Formatted at 2026-06-20 09:26:04 on dist-test-slave-706q"
server_key: "a231aac6156b7bdedd34b510a4b5202e"
server_key_iv: "27ae32f88ad61da3f5c33887f5099e78"
server_key_version: "encryptionkey@0"
I20260620 09:26:04.615311 16806 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781947562949448-16558-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781947562949448-16558-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781947562949448-16558-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:26:04.661234 16806 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:26:04.663975 16806 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:26:04.664134 16806 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:26:04.664436 16806 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:26:04.664827 16806 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:26:04.664862 16806 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:04.664908 16806 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:26:04.664935 16806 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:04.671177 16806 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.130:35955
I20260620 09:26:04.671221 16934 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.130:35955 every 8 connection(s)
I20260620 09:26:04.671624 16806 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781947562949448-16558-0/minicluster-data/ts-1/data/info.pb
Jun 20 09:26:04 dist-test-slave-706q krb5kdc[16565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947564, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.130@KRBTEST.COM for kudu/127.16.43.190@KRBTEST.COM
I20260620 09:26:04.680963 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 16806
I20260620 09:26:04.681082 16558 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781947562949448-16558-0/minicluster-data/ts-1/wal/instance
I20260620 09:26:04.681154 16668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:04.672474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.130:56547 (local address 127.16.43.190:44389)
0620 09:26:04.672552 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:04.672554 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:04.672796 (+ 242us) server_negotiation.cc:407] Connection header received
0620 09:26:04.673298 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:04.673299 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:04.673310 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:04.673363 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:04.674270 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:04.674813 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:04.675952 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:04.676146 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:04.676220 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:04.678380 (+ 2160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:04.678382 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:04.678382 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:26:04.678389 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:04.679832 (+ 1443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:04.680206 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:04.680207 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:04.680207 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:04.680261 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:04.680520 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:04.680521 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:04.680521 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:04.680626 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:04.680677 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:04.680953 (+ 276us) server_negotiation.cc:299] Negotiation successful
0620 09:26:04.681059 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 09:26:04.681306 16558 external_mini_cluster.cc:1428] Setting key 881b80ec3f4151f4f71e9f3a8e9f0a04
I20260620 09:26:04.681392 16937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:04.672451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:44389 (local address 127.16.43.130:56547)
0620 09:26:04.672689 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:04.672726 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:04.673213 (+ 487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:04.673425 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:04.673428 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:04.673983 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:04.674188 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:04.674193 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:04.674901 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:04.674902 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:04.675865 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:04.675868 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:04.676227 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:04.676228 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:04.676325 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:04.676818 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:26:04.676827 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:26:04.678283 (+ 1456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:26:04.679903 (+ 1620us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:04.679910 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:04.679911 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:04.680139 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:04.680321 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:04.680322 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:04.680322 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:04.680433 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:04.680682 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:26:04.680687 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:26:04.680803 (+ 116us) client_negotiation.cc:769] Sending connection context
0620 09:26:04.680926 (+ 123us) client_negotiation.cc:240] Negotiation successful
0620 09:26:04.681118 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":71,"threads_started":1}
I20260620 09:26:04.681877 16935 heartbeater.cc:347] Connected to a master server at 127.16.43.190:44389
I20260620 09:26:04.681975 16935 heartbeater.cc:464] Registering TS with master...
I20260620 09:26:04.682163 16935 heartbeater.cc:511] Master 127.16.43.190:44389 requested a full tablet report, sending...
I20260620 09:26:04.682595 16609 ts_manager.cc:194] Registered new tserver with Master: 84fe1c76645d4914b4f0231dc0a7c9a2 (127.16.43.130:35955)
I20260620 09:26:04.683099 16609 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.43.130@KRBTEST.COM'} at 127.16.43.130:56547
WARNING: no policy specified for kudu/127.16.43.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.43.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.43.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.131.keytab.
Entry for principal kudu/127.16.43.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.131.keytab.
I20260620 09:26:04.706755 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781947562949448-16558-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781947562949448-16558-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781947562949448-16558-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781947562949448-16558-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.43.131:0
--local_ip_for_outbound_sockets=127.16.43.131
--webserver_interface=127.16.43.131
--webserver_port=0
--tserver_master_addrs=127.16.43.190:44389
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.131.keytab
--principal=kudu/127.16.43.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.43.148:37453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781947562949448-16558-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781947562949448-16558-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781947562949448-16558-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:26:04.788765 16940 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:26:04.788982 16940 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:26:04.789014 16940 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:26:04.790216 16940 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:26:04.790262 16940 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:26:04.790290 16940 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.43.131
I20260620 09:26:04.791795 16940 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:37453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781947562949448-16558-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781947562949448-16558-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.131.keytab
--principal=kudu/127.16.43.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.43.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781947562949448-16558-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.43.131
--webserver_port=0
--tserver_master_addrs=127.16.43.190:44389
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16940
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.43.131
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781947562949448-16558-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:26:04.792049 16940 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:26:04.792284 16940 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:26:04.797791 16948 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:26:04.797827 16945 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:26:04.797832 16946 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:26:04.798048 16940 server_base.cc:1034] running on GCE node
I20260620 09:26:04.798272 16940 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:26:04.798512 16940 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:26:04.799666 16940 hybrid_clock.cc:648] HybridClock initialized: now 1781947564799655 us; error 32 us; skew 500 ppm
Jun 20 09:26:04 dist-test-slave-706q krb5kdc[16565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947564, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:26:04.802665 16940 init.cc:377] Logged in from keytab as kudu/127.16.43.131@KRBTEST.COM (short username kudu)
I20260620 09:26:04.803109 16940 webserver.cc:460] Webserver started at http://127.16.43.131:39813/ using document root <none> and password file <none>
I20260620 09:26:04.803334 16940 fs_manager.cc:359] Metadata directory not provided
I20260620 09:26:04.803401 16940 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:26:04.803560 16940 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:26:04.804610 16940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781947562949448-16558-0/minicluster-data/ts-2/data/instance:
uuid: "f8804a81a24c485e843b5906ac1ab268"
format_stamp: "Formatted at 2026-06-20 09:26:04 on dist-test-slave-706q"
server_key: "4e1275aaf8dd265f28c553b470a18761"
server_key_iv: "3808151757cc7798d750e57cccf984a4"
server_key_version: "encryptionkey@0"
I20260620 09:26:04.804986 16940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781947562949448-16558-0/minicluster-data/ts-2/wal/instance:
uuid: "f8804a81a24c485e843b5906ac1ab268"
format_stamp: "Formatted at 2026-06-20 09:26:04 on dist-test-slave-706q"
server_key: "4e1275aaf8dd265f28c553b470a18761"
server_key_iv: "3808151757cc7798d750e57cccf984a4"
server_key_version: "encryptionkey@0"
I20260620 09:26:04.806326 16940 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 09:26:04.807170 16955 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:04.807334 16940 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 09:26:04.807441 16940 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781947562949448-16558-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781947562949448-16558-0/minicluster-data/ts-2/wal
uuid: "f8804a81a24c485e843b5906ac1ab268"
format_stamp: "Formatted at 2026-06-20 09:26:04 on dist-test-slave-706q"
server_key: "4e1275aaf8dd265f28c553b470a18761"
server_key_iv: "3808151757cc7798d750e57cccf984a4"
server_key_version: "encryptionkey@0"
I20260620 09:26:04.807519 16940 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781947562949448-16558-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781947562949448-16558-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781947562949448-16558-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:26:04.822571 16940 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:26:04.824774 16940 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:26:04.824949 16940 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:26:04.825207 16940 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:26:04.825662 16940 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:26:04.825706 16940 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:04.825740 16940 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:26:04.825771 16940 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:04.831626 16940 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.131:37191
I20260620 09:26:04.831684 17068 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.131:37191 every 8 connection(s)
I20260620 09:26:04.832077 16940 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781947562949448-16558-0/minicluster-data/ts-2/data/info.pb
Jun 20 09:26:04 dist-test-slave-706q krb5kdc[16565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947564, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.131@KRBTEST.COM for kudu/127.16.43.190@KRBTEST.COM
I20260620 09:26:04.841955 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 16940
I20260620 09:26:04.841970 16668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:04.832881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.131:57083 (local address 127.16.43.190:44389)
0620 09:26:04.832995 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:04.832998 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:04.833207 (+ 209us) server_negotiation.cc:407] Connection header received
0620 09:26:04.833774 (+ 567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:04.833775 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:04.833787 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:04.833838 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:04.834808 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:04.835316 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:04.836443 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:04.836642 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:04.836729 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:04.839217 (+ 2488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:04.839218 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:04.839218 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:26:04.839229 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:04.840707 (+ 1478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:04.841083 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:04.841084 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:04.841084 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:04.841132 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:04.841332 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:04.841332 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:04.841333 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:04.841433 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:04.841484 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:04.841744 (+ 260us) server_negotiation.cc:299] Negotiation successful
0620 09:26:04.841871 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 09:26:04.841996 17071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:04.832857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:44389 (local address 127.16.43.131:57083)
0620 09:26:04.833065 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:04.833118 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:04.833661 (+ 543us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:04.833920 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:04.833924 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:04.834492 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:04.834705 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:04.834710 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:04.835412 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:04.835414 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:04.836333 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:04.836336 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:04.836738 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:04.836740 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:04.836879 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:04.837425 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:26:04.837434 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:26:04.839094 (+ 1660us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:26:04.840792 (+ 1698us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:04.840797 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:04.840799 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:04.841020 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:04.841189 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:04.841189 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:04.841189 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:04.841281 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:04.841487 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:26:04.841491 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:26:04.841604 (+ 113us) client_negotiation.cc:769] Sending connection context
0620 09:26:04.841686 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 09:26:04.841800 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":92,"threads_started":1}
I20260620 09:26:04.842072 16558 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781947562949448-16558-0/minicluster-data/ts-2/wal/instance
I20260620 09:26:04.842242 16558 external_mini_cluster.cc:1428] Setting key 64385f80d2f70c7502ef799e5a8bad4b
I20260620 09:26:04.842825 17069 heartbeater.cc:347] Connected to a master server at 127.16.43.190:44389
I20260620 09:26:04.842927 17069 heartbeater.cc:464] Registering TS with master...
I20260620 09:26:04.843127 17069 heartbeater.cc:511] Master 127.16.43.190:44389 requested a full tablet report, sending...
I20260620 09:26:04.843544 16610 ts_manager.cc:194] Registered new tserver with Master: f8804a81a24c485e843b5906ac1ab268 (127.16.43.131:37191)
I20260620 09:26:04.843899 16610 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.43.131@KRBTEST.COM'} at 127.16.43.131:57083
I20260620 09:26:04.854614 16558 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 09:26:04 dist-test-slave-706q krb5kdc[16565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947564, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 09:26:04 dist-test-slave-706q krb5kdc[16565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947564, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.43.129@KRBTEST.COM
I20260620 09:26:04.879045 17076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:04.870373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42178 (local address 127.16.43.129:44079)
0620 09:26:04.870602 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:04.870608 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:04.870649 (+ 41us) server_negotiation.cc:407] Connection header received
0620 09:26:04.870675 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:04.870677 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:04.870691 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:04.870751 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:04.871361 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:04.872285 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:04.873046 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:04.873394 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:04.873461 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:04.876200 (+ 2739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:04.876208 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:04.876208 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:26:04.876215 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:04.877832 (+ 1617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:04.878181 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:04.878184 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:04.878184 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:04.878240 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:04.878474 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:04.878475 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:04.878475 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:04.878622 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:04.878756 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:04.878843 (+ 87us) server_negotiation.cc:299] Negotiation successful
0620 09:26:04.878905 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":96,"threads_started":1}
W20260620 09:26:04.879571 16715 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42178
Jun 20 09:26:04 dist-test-slave-706q krb5kdc[16565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947564, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 09:26:04 dist-test-slave-706q krb5kdc[16565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947564, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.43.129@KRBTEST.COM
I20260620 09:26:04.903198 17076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:04.895462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42188 (local address 127.16.43.129:44079)
0620 09:26:04.895581 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:04.895584 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:04.895716 (+ 132us) server_negotiation.cc:407] Connection header received
0620 09:26:04.895788 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:04.895789 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:04.895803 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:04.895869 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:04.896648 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:04.897148 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:04.897913 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:04.898120 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:04.898177 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:04.900676 (+ 2499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:04.900678 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:04.900678 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:26:04.900686 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:04.902064 (+ 1378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:04.902400 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:04.902401 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:04.902401 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:04.902520 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:04.902731 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:04.902733 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:04.902733 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:04.902839 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:04.902940 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:04.903048 (+ 108us) server_negotiation.cc:299] Negotiation successful
0620 09:26:04.903128 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260620 09:26:04.903952 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 16672
I20260620 09:26:04.910424 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 16806
I20260620 09:26:04.915457 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 16940
I20260620 09:26:04.920243 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 16578
2026-06-20T09:26:04Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1979 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781947562949448-16558-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:26:04 dist-test-slave-706q krb5kdc[17083](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:26:04 dist-test-slave-706q krb5kdc[17083](info): set up 2 sockets
Jun 20 09:26:04 dist-test-slave-706q krb5kdc[17083](info): commencing operation
krb5kdc: starting...
W20260620 09:26:06.020174 16558 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781947562949448-16558-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781947562949448-16558-0/krb5kdc/test-user.keytab.
Jun 20 09:26:06 dist-test-slave-706q krb5kdc[17083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947566, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:26:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:26:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.43.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.43.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.43.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.190.keytab.
Entry for principal kudu/127.16.43.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.190.keytab.
I20260620 09:26:06.143422 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781947562949448-16558-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781947562949448-16558-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781947562949448-16558-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.43.190:36723
--webserver_interface=127.16.43.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.190.keytab
--principal=kudu/127.16.43.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.43.148:44873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.43.190:36723
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781947562949448-16558-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781947562949448-16558-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781947562949448-16558-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:26:06.227206 17096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:26:06.227432 17096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:26:06.227455 17096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:26:06.228694 17096 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:26:06.228744 17096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:26:06.228770 17096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:26:06.228785 17096 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:26:06.228797 17096 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:26:06.230247 17096 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:44873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781947562949448-16558-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781947562949448-16558-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.43.190:36723
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.190.keytab
--principal=kudu/127.16.43.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.43.190:36723
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.43.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17096
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781947562949448-16558-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:26:06.230448 17096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:26:06.230643 17096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:26:06.235698 17104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:26:06.235700 17101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:26:06.235862 17096 server_base.cc:1034] running on GCE node
W20260620 09:26:06.235986 17102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:26:06.236274 17096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:26:06.236519 17096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:26:06.237699 17096 hybrid_clock.cc:648] HybridClock initialized: now 1781947566237665 us; error 45 us; skew 500 ppm
Jun 20 09:26:06 dist-test-slave-706q krb5kdc[17083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947566, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:26:06.244187 17096 init.cc:377] Logged in from keytab as kudu/127.16.43.190@KRBTEST.COM (short username kudu)
I20260620 09:26:06.244686 17096 webserver.cc:460] Webserver started at http://127.16.43.190:38453/ using document root <none> and password file <none>
I20260620 09:26:06.244935 17096 fs_manager.cc:359] Metadata directory not provided
I20260620 09:26:06.245004 17096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:26:06.245159 17096 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:26:06.246255 17096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781947562949448-16558-0/minicluster-data/master-0/data/instance:
uuid: "8244f746730b4ac6af3748903e0aae83"
format_stamp: "Formatted at 2026-06-20 09:26:06 on dist-test-slave-706q"
server_key: "d3d8ab48ab69dc95d394d44a8da11359"
server_key_iv: "a6e75c6dbac1b2743f210a25ed9dea34"
server_key_version: "encryptionkey@0"
I20260620 09:26:06.246628 17096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781947562949448-16558-0/minicluster-data/master-0/wal/instance:
uuid: "8244f746730b4ac6af3748903e0aae83"
format_stamp: "Formatted at 2026-06-20 09:26:06 on dist-test-slave-706q"
server_key: "d3d8ab48ab69dc95d394d44a8da11359"
server_key_iv: "a6e75c6dbac1b2743f210a25ed9dea34"
server_key_version: "encryptionkey@0"
I20260620 09:26:06.248076 17096 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 09:26:06.248868 17111 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:06.249068 17096 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:26:06.249176 17096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781947562949448-16558-0/minicluster-data/master-0/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781947562949448-16558-0/minicluster-data/master-0/wal
uuid: "8244f746730b4ac6af3748903e0aae83"
format_stamp: "Formatted at 2026-06-20 09:26:06 on dist-test-slave-706q"
server_key: "d3d8ab48ab69dc95d394d44a8da11359"
server_key_iv: "a6e75c6dbac1b2743f210a25ed9dea34"
server_key_version: "encryptionkey@0"
I20260620 09:26:06.249248 17096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781947562949448-16558-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781947562949448-16558-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781947562949448-16558-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:26:06.273820 17096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:26:06.276137 17096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:26:06.276309 17096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:26:06.280339 17096 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.190:36723
I20260620 09:26:06.280395 17163 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.190:36723 every 8 connection(s)
I20260620 09:26:06.280767 17096 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
I20260620 09:26:06.281693 17164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:26:06.284083 17164 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8244f746730b4ac6af3748903e0aae83: Bootstrap starting.
I20260620 09:26:06.284943 17164 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8244f746730b4ac6af3748903e0aae83: Neither blocks nor log segments found. Creating new log.
I20260620 09:26:06.285286 17164 log.cc:824] T 00000000000000000000000000000000 P 8244f746730b4ac6af3748903e0aae83: Log is configured to *not* fsync() on all Append() calls
I20260620 09:26:06.286118 17164 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8244f746730b4ac6af3748903e0aae83: No bootstrap required, opened a new log
I20260620 09:26:06.287941 17164 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8244f746730b4ac6af3748903e0aae83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8244f746730b4ac6af3748903e0aae83" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 36723 } }
I20260620 09:26:06.288079 17164 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8244f746730b4ac6af3748903e0aae83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:26:06.288102 17164 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8244f746730b4ac6af3748903e0aae83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8244f746730b4ac6af3748903e0aae83, State: Initialized, Role: FOLLOWER
I20260620 09:26:06.288277 17164 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8244f746730b4ac6af3748903e0aae83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8244f746730b4ac6af3748903e0aae83" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 36723 } }
I20260620 09:26:06.288353 17164 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8244f746730b4ac6af3748903e0aae83 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:26:06.288374 17164 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8244f746730b4ac6af3748903e0aae83 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:26:06.288425 17164 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8244f746730b4ac6af3748903e0aae83 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:26:06.288785 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 17096
I20260620 09:26:06.288892 16558 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781947562949448-16558-0/minicluster-data/master-0/wal/instance
I20260620 09:26:06.289063 16558 external_mini_cluster.cc:1428] Setting key f9f281628143f6bff9befe60a78b3973
I20260620 09:26:06.289119 17164 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8244f746730b4ac6af3748903e0aae83 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8244f746730b4ac6af3748903e0aae83" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 36723 } }
I20260620 09:26:06.289279 17164 leader_election.cc:302] T 00000000000000000000000000000000 P 8244f746730b4ac6af3748903e0aae83 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8244f746730b4ac6af3748903e0aae83; no voters:
I20260620 09:26:06.289479 17164 leader_election.cc:288] T 00000000000000000000000000000000 P 8244f746730b4ac6af3748903e0aae83 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:26:06.289508 17167 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8244f746730b4ac6af3748903e0aae83 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:26:06.289625 17167 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8244f746730b4ac6af3748903e0aae83 [term 1 LEADER]: Becoming Leader. State: Replica: 8244f746730b4ac6af3748903e0aae83, State: Running, Role: LEADER
I20260620 09:26:06.289721 17167 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8244f746730b4ac6af3748903e0aae83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8244f746730b4ac6af3748903e0aae83" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 36723 } }
I20260620 09:26:06.289917 17164 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8244f746730b4ac6af3748903e0aae83 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:26:06.290053 17167 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8244f746730b4ac6af3748903e0aae83 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8244f746730b4ac6af3748903e0aae83" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8244f746730b4ac6af3748903e0aae83" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 36723 } } }
I20260620 09:26:06.290144 17167 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8244f746730b4ac6af3748903e0aae83 [sys.catalog]: This master's current role is: LEADER
I20260620 09:26:06.290242 17171 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8244f746730b4ac6af3748903e0aae83 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8244f746730b4ac6af3748903e0aae83. Latest consensus state: current_term: 1 leader_uuid: "8244f746730b4ac6af3748903e0aae83" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8244f746730b4ac6af3748903e0aae83" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 36723 } } }
I20260620 09:26:06.290325 17173 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:26:06.290338 17171 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8244f746730b4ac6af3748903e0aae83 [sys.catalog]: This master's current role is: LEADER
I20260620 09:26:06.290933 17173 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:26:06.292824 17173 catalog_manager.cc:1347] Generated new cluster ID: 13c951af03cf445bb07d36045e9fc9d5
I20260620 09:26:06.292881 17173 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 09:26:06 dist-test-slave-706q krb5kdc[17083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947566, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.43.190@KRBTEST.COM
I20260620 09:26:06.301355 17170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:06.289609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38684 (local address 127.16.43.190:36723)
0620 09:26:06.290985 (+ 1376us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:06.290995 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:06.291029 (+ 34us) server_negotiation.cc:407] Connection header received
0620 09:26:06.291224 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:06.291232 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:06.291312 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:06.292204 (+ 892us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:06.292303 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:06.293453 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:06.294309 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:06.294753 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:06.294876 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:06.297409 (+ 2533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:06.297414 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:06.297414 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:26:06.297426 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:06.299711 (+ 2285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:06.300095 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:06.300104 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:06.300104 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:06.300229 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:06.300448 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:06.300449 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:06.300450 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:06.300702 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:06.300872 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:06.300971 (+ 99us) server_negotiation.cc:299] Negotiation successful
0620 09:26:06.301126 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1330,"thread_start_us":59,"threads_started":1}
I20260620 09:26:06.301505 17173 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:26:06.302134 17173 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:26:06.308974 17173 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8244f746730b4ac6af3748903e0aae83: Generated new TSK 0
I20260620 09:26:06.309226 17173 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.43.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.43.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.43.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.129.keytab.
Entry for principal kudu/127.16.43.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.129.keytab.
I20260620 09:26:06.340875 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781947562949448-16558-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781947562949448-16558-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781947562949448-16558-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781947562949448-16558-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.43.129:0
--local_ip_for_outbound_sockets=127.16.43.129
--webserver_interface=127.16.43.129
--webserver_port=0
--tserver_master_addrs=127.16.43.190:36723
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.129.keytab
--principal=kudu/127.16.43.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.43.148:44873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781947562949448-16558-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781947562949448-16558-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781947562949448-16558-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:26:06.459517 17190 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:26:06.459846 17190 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:26:06.459887 17190 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:26:06.461658 17190 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:26:06.461720 17190 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:26:06.461761 17190 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.43.129
I20260620 09:26:06.464066 17190 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:44873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781947562949448-16558-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781947562949448-16558-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.129.keytab
--principal=kudu/127.16.43.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.43.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781947562949448-16558-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.43.129
--webserver_port=0
--tserver_master_addrs=127.16.43.190:36723
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17190
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.43.129
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781947562949448-16558-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:26:06.464394 17190 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:26:06.464743 17190 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:26:06.470736 17195 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:26:06.470683 17198 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:26:06.470892 17190 server_base.cc:1034] running on GCE node
W20260620 09:26:06.471004 17196 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:26:06.471182 17190 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:26:06.471448 17190 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:26:06.472615 17190 hybrid_clock.cc:648] HybridClock initialized: now 1781947566472597 us; error 54 us; skew 500 ppm
Jun 20 09:26:06 dist-test-slave-706q krb5kdc[17083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947566, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:26:06.475740 17190 init.cc:377] Logged in from keytab as kudu/127.16.43.129@KRBTEST.COM (short username kudu)
I20260620 09:26:06.476269 17190 webserver.cc:460] Webserver started at http://127.16.43.129:37699/ using document root <none> and password file <none>
I20260620 09:26:06.476550 17190 fs_manager.cc:359] Metadata directory not provided
I20260620 09:26:06.476627 17190 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:26:06.476787 17190 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:26:06.477994 17190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781947562949448-16558-0/minicluster-data/ts-0/data/instance:
uuid: "b40e28324fef4972b986d2af3583add8"
format_stamp: "Formatted at 2026-06-20 09:26:06 on dist-test-slave-706q"
server_key: "a9799aaa19a6f6d22967eef88a5afbfe"
server_key_iv: "70dd1f657d98f3e3841133562335f677"
server_key_version: "encryptionkey@0"
I20260620 09:26:06.478375 17190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781947562949448-16558-0/minicluster-data/ts-0/wal/instance:
uuid: "b40e28324fef4972b986d2af3583add8"
format_stamp: "Formatted at 2026-06-20 09:26:06 on dist-test-slave-706q"
server_key: "a9799aaa19a6f6d22967eef88a5afbfe"
server_key_iv: "70dd1f657d98f3e3841133562335f677"
server_key_version: "encryptionkey@0"
I20260620 09:26:06.479853 17190 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 09:26:06.480736 17205 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:06.480922 17190 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:26:06.481017 17190 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781947562949448-16558-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781947562949448-16558-0/minicluster-data/ts-0/wal
uuid: "b40e28324fef4972b986d2af3583add8"
format_stamp: "Formatted at 2026-06-20 09:26:06 on dist-test-slave-706q"
server_key: "a9799aaa19a6f6d22967eef88a5afbfe"
server_key_iv: "70dd1f657d98f3e3841133562335f677"
server_key_version: "encryptionkey@0"
I20260620 09:26:06.481103 17190 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781947562949448-16558-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781947562949448-16558-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781947562949448-16558-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:26:06.492103 17190 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:26:06.494803 17190 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:26:06.495003 17190 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:26:06.495270 17190 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:26:06.495652 17190 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:26:06.495708 17190 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:06.495750 17190 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:26:06.495776 17190 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:06.501919 17190 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.129:45599
I20260620 09:26:06.501961 17318 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.129:45599 every 8 connection(s)
I20260620 09:26:06.502290 17190 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781947562949448-16558-0/minicluster-data/ts-0/data/info.pb
I20260620 09:26:06.506973 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 17190
I20260620 09:26:06.507089 16558 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781947562949448-16558-0/minicluster-data/ts-0/wal/instance
I20260620 09:26:06.507241 16558 external_mini_cluster.cc:1428] Setting key 8353b080338cdcf8034dc4d2a070d1d4
Jun 20 09:26:06 dist-test-slave-706q krb5kdc[17083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947566, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.129@KRBTEST.COM for kudu/127.16.43.190@KRBTEST.COM
I20260620 09:26:06.512195 17170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:06.503061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.129:45641 (local address 127.16.43.190:36723)
0620 09:26:06.503121 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:06.503124 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:06.503338 (+ 214us) server_negotiation.cc:407] Connection header received
0620 09:26:06.504387 (+ 1049us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:06.504389 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:06.504405 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:06.504457 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:06.505110 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:06.505644 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:06.506669 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:06.506891 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:06.506979 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:06.509236 (+ 2257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:06.509238 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:06.509238 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:26:06.509246 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:06.510689 (+ 1443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:06.511176 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:06.511177 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:06.511178 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:06.511238 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:06.511490 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:06.511491 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:06.511491 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:06.511659 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:06.511737 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:06.512011 (+ 274us) server_negotiation.cc:299] Negotiation successful
0620 09:26:06.512104 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 09:26:06.512418 17321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:06.503065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:36723 (local address 127.16.43.129:45641)
0620 09:26:06.503210 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:06.503248 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:06.504246 (+ 998us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:06.504535 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:06.504539 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:06.504798 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:06.504990 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:06.504994 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:06.505745 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:06.505747 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:06.506562 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:06.506564 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:06.506989 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:06.506990 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:06.507143 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:06.507744 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:26:06.507753 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:26:06.509140 (+ 1387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:26:06.510789 (+ 1649us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:06.510797 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:06.510799 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:06.511092 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:06.511304 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:06.511305 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:06.511306 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:06.511422 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:06.511728 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:26:06.511735 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:26:06.511877 (+ 142us) client_negotiation.cc:769] Sending connection context
0620 09:26:06.511989 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 09:26:06.512146 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":79,"threads_started":1}
I20260620 09:26:06.512959 17319 heartbeater.cc:347] Connected to a master server at 127.16.43.190:36723
I20260620 09:26:06.513068 17319 heartbeater.cc:464] Registering TS with master...
I20260620 09:26:06.513280 17319 heartbeater.cc:511] Master 127.16.43.190:36723 requested a full tablet report, sending...
I20260620 09:26:06.513876 17128 ts_manager.cc:194] Registered new tserver with Master: b40e28324fef4972b986d2af3583add8 (127.16.43.129:45599)
I20260620 09:26:06.514873 17128 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.43.129@KRBTEST.COM'} at 127.16.43.129:45641
WARNING: no policy specified for kudu/127.16.43.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.43.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.43.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.130.keytab.
Entry for principal kudu/127.16.43.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.130.keytab.
I20260620 09:26:06.534698 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781947562949448-16558-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781947562949448-16558-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781947562949448-16558-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781947562949448-16558-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.43.130:0
--local_ip_for_outbound_sockets=127.16.43.130
--webserver_interface=127.16.43.130
--webserver_port=0
--tserver_master_addrs=127.16.43.190:36723
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.130.keytab
--principal=kudu/127.16.43.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.43.148:44873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781947562949448-16558-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781947562949448-16558-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781947562949448-16558-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:26:06.621254 17324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:26:06.621474 17324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:26:06.621500 17324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:26:06.622655 17324 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:26:06.622699 17324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:26:06.622728 17324 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.43.130
I20260620 09:26:06.624231 17324 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:44873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781947562949448-16558-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781947562949448-16558-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.130.keytab
--principal=kudu/127.16.43.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.43.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781947562949448-16558-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.43.130
--webserver_port=0
--tserver_master_addrs=127.16.43.190:36723
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17324
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.43.130
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781947562949448-16558-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:26:06.624488 17324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:26:06.624734 17324 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:26:06.630167 17329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:26:06.630179 17330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:26:06.630179 17332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:26:06.630235 17324 server_base.cc:1034] running on GCE node
I20260620 09:26:06.630625 17324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:26:06.630901 17324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:26:06.632119 17324 hybrid_clock.cc:648] HybridClock initialized: now 1781947566632100 us; error 31 us; skew 500 ppm
Jun 20 09:26:06 dist-test-slave-706q krb5kdc[17083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947566, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:26:06.634723 17324 init.cc:377] Logged in from keytab as kudu/127.16.43.130@KRBTEST.COM (short username kudu)
I20260620 09:26:06.635182 17324 webserver.cc:460] Webserver started at http://127.16.43.130:38715/ using document root <none> and password file <none>
I20260620 09:26:06.635408 17324 fs_manager.cc:359] Metadata directory not provided
I20260620 09:26:06.635478 17324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:26:06.635658 17324 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:26:06.636718 17324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781947562949448-16558-0/minicluster-data/ts-1/data/instance:
uuid: "fa50fddf5dd14777ac8bc22dae2c058c"
format_stamp: "Formatted at 2026-06-20 09:26:06 on dist-test-slave-706q"
server_key: "15ddcc4e863c8c0a8912308686244759"
server_key_iv: "437c9655dc089e86d5cc9cd04775d926"
server_key_version: "encryptionkey@0"
I20260620 09:26:06.637082 17324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781947562949448-16558-0/minicluster-data/ts-1/wal/instance:
uuid: "fa50fddf5dd14777ac8bc22dae2c058c"
format_stamp: "Formatted at 2026-06-20 09:26:06 on dist-test-slave-706q"
server_key: "15ddcc4e863c8c0a8912308686244759"
server_key_iv: "437c9655dc089e86d5cc9cd04775d926"
server_key_version: "encryptionkey@0"
I20260620 09:26:06.638420 17324 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 09:26:06.639348 17339 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:06.639559 17324 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 09:26:06.639647 17324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781947562949448-16558-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781947562949448-16558-0/minicluster-data/ts-1/wal
uuid: "fa50fddf5dd14777ac8bc22dae2c058c"
format_stamp: "Formatted at 2026-06-20 09:26:06 on dist-test-slave-706q"
server_key: "15ddcc4e863c8c0a8912308686244759"
server_key_iv: "437c9655dc089e86d5cc9cd04775d926"
server_key_version: "encryptionkey@0"
I20260620 09:26:06.639756 17324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781947562949448-16558-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781947562949448-16558-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781947562949448-16558-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:26:06.655256 17324 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:26:06.657639 17324 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:26:06.657815 17324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:26:06.658061 17324 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:26:06.658381 17324 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:26:06.658437 17324 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:06.658476 17324 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:26:06.658504 17324 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:06.664407 17324 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.130:45699
I20260620 09:26:06.664453 17452 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.130:45699 every 8 connection(s)
I20260620 09:26:06.664820 17324 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781947562949448-16558-0/minicluster-data/ts-1/data/info.pb
I20260620 09:26:06.669623 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 17324
I20260620 09:26:06.669739 16558 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781947562949448-16558-0/minicluster-data/ts-1/wal/instance
I20260620 09:26:06.669931 16558 external_mini_cluster.cc:1428] Setting key 3ff7e664ac16a620a3381aacac0e6d73
Jun 20 09:26:06 dist-test-slave-706q krb5kdc[17083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947566, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.130@KRBTEST.COM for kudu/127.16.43.190@KRBTEST.COM
I20260620 09:26:06.675223 17170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:06.665682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.130:57615 (local address 127.16.43.190:36723)
0620 09:26:06.665768 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:06.665771 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:06.665867 (+ 96us) server_negotiation.cc:407] Connection header received
0620 09:26:06.666810 (+ 943us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:06.666812 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:06.666824 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:06.666891 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:06.667492 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:06.668137 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:06.669159 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:06.669355 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:06.669436 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:06.672260 (+ 2824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:06.672261 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:06.672262 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:26:06.672271 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:06.673725 (+ 1454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:06.674182 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:06.674183 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:06.674183 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:06.674249 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:06.674523 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:06.674524 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:06.674525 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:06.674659 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:06.674737 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:06.675029 (+ 292us) server_negotiation.cc:299] Negotiation successful
0620 09:26:06.675119 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 09:26:06.675453 17455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:06.665558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:36723 (local address 127.16.43.130:57615)
0620 09:26:06.665720 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:06.665771 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:06.666692 (+ 921us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:06.666947 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:06.666950 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:06.667203 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:06.667396 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:06.667400 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:06.668227 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:06.668228 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:06.669061 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:06.669063 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:06.669465 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:06.669466 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:06.669625 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:06.670329 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:26:06.670339 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:26:06.672156 (+ 1817us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:26:06.673836 (+ 1680us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:06.673844 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:06.673846 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:06.674100 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:06.674318 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:06.674319 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:06.674320 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:06.674452 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:06.674728 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:26:06.674736 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:26:06.674895 (+ 159us) client_negotiation.cc:769] Sending connection context
0620 09:26:06.675010 (+ 115us) client_negotiation.cc:240] Negotiation successful
0620 09:26:06.675196 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":90,"threads_started":1}
I20260620 09:26:06.675961 17453 heartbeater.cc:347] Connected to a master server at 127.16.43.190:36723
I20260620 09:26:06.676070 17453 heartbeater.cc:464] Registering TS with master...
I20260620 09:26:06.676293 17453 heartbeater.cc:511] Master 127.16.43.190:36723 requested a full tablet report, sending...
I20260620 09:26:06.676688 17128 ts_manager.cc:194] Registered new tserver with Master: fa50fddf5dd14777ac8bc22dae2c058c (127.16.43.130:45699)
I20260620 09:26:06.677103 17128 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.43.130@KRBTEST.COM'} at 127.16.43.130:57615
WARNING: no policy specified for kudu/127.16.43.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.43.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.43.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.131.keytab.
Entry for principal kudu/127.16.43.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.131.keytab.
I20260620 09:26:06.698369 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781947562949448-16558-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781947562949448-16558-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781947562949448-16558-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781947562949448-16558-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.43.131:0
--local_ip_for_outbound_sockets=127.16.43.131
--webserver_interface=127.16.43.131
--webserver_port=0
--tserver_master_addrs=127.16.43.190:36723
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.131.keytab
--principal=kudu/127.16.43.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.43.148:44873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781947562949448-16558-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781947562949448-16558-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781947562949448-16558-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:26:06.785759 17458 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:26:06.785993 17458 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:26:06.786019 17458 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:26:06.787207 17458 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:26:06.787248 17458 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:26:06.787276 17458 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.43.131
I20260620 09:26:06.788830 17458 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:44873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781947562949448-16558-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781947562949448-16558-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.131.keytab
--principal=kudu/127.16.43.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.43.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781947562949448-16558-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.43.131
--webserver_port=0
--tserver_master_addrs=127.16.43.190:36723
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17458
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.43.131
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781947562949448-16558-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:26:06.789078 17458 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:26:06.789342 17458 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:26:06.794987 17464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:26:06.795082 17466 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:26:06.795009 17463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:26:06.795212 17458 server_base.cc:1034] running on GCE node
I20260620 09:26:06.795418 17458 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:26:06.795686 17458 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:26:06.796815 17458 hybrid_clock.cc:648] HybridClock initialized: now 1781947566796793 us; error 39 us; skew 500 ppm
Jun 20 09:26:06 dist-test-slave-706q krb5kdc[17083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947566, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:26:06.799413 17458 init.cc:377] Logged in from keytab as kudu/127.16.43.131@KRBTEST.COM (short username kudu)
I20260620 09:26:06.799906 17458 webserver.cc:460] Webserver started at http://127.16.43.131:44687/ using document root <none> and password file <none>
I20260620 09:26:06.800140 17458 fs_manager.cc:359] Metadata directory not provided
I20260620 09:26:06.800207 17458 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:26:06.800350 17458 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:26:06.801416 17458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781947562949448-16558-0/minicluster-data/ts-2/data/instance:
uuid: "5c1546eafa874d78aff0d958f467b5cb"
format_stamp: "Formatted at 2026-06-20 09:26:06 on dist-test-slave-706q"
server_key: "9246967675835f0d3e5baf9b2e0a5809"
server_key_iv: "95ea0aef3fa87414c3ff429ff96a2731"
server_key_version: "encryptionkey@0"
I20260620 09:26:06.801776 17458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781947562949448-16558-0/minicluster-data/ts-2/wal/instance:
uuid: "5c1546eafa874d78aff0d958f467b5cb"
format_stamp: "Formatted at 2026-06-20 09:26:06 on dist-test-slave-706q"
server_key: "9246967675835f0d3e5baf9b2e0a5809"
server_key_iv: "95ea0aef3fa87414c3ff429ff96a2731"
server_key_version: "encryptionkey@0"
I20260620 09:26:06.803133 17458 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 09:26:06.804000 17473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:06.804229 17458 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 09:26:06.804342 17458 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781947562949448-16558-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781947562949448-16558-0/minicluster-data/ts-2/wal
uuid: "5c1546eafa874d78aff0d958f467b5cb"
format_stamp: "Formatted at 2026-06-20 09:26:06 on dist-test-slave-706q"
server_key: "9246967675835f0d3e5baf9b2e0a5809"
server_key_iv: "95ea0aef3fa87414c3ff429ff96a2731"
server_key_version: "encryptionkey@0"
I20260620 09:26:06.804430 17458 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781947562949448-16558-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781947562949448-16558-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781947562949448-16558-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:26:06.824105 17458 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:26:06.826848 17458 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:26:06.827034 17458 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:26:06.827332 17458 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:26:06.827744 17458 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:26:06.827811 17458 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:06.827850 17458 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:26:06.827881 17458 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:06.834322 17458 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.131:37139
I20260620 09:26:06.834362 17586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.131:37139 every 8 connection(s)
I20260620 09:26:06.834730 17458 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781947562949448-16558-0/minicluster-data/ts-2/data/info.pb
Jun 20 09:26:06 dist-test-slave-706q krb5kdc[17083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947566, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.131@KRBTEST.COM for kudu/127.16.43.190@KRBTEST.COM
I20260620 09:26:06.844044 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 17458
I20260620 09:26:06.844141 16558 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781947562949448-16558-0/minicluster-data/ts-2/wal/instance
I20260620 09:26:06.844326 16558 external_mini_cluster.cc:1428] Setting key b86cbc5c5fa97527147185b104207223
I20260620 09:26:06.844508 17170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:06.835634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.131:34549 (local address 127.16.43.190:36723)
0620 09:26:06.835691 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:06.835694 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:06.835913 (+ 219us) server_negotiation.cc:407] Connection header received
0620 09:26:06.836911 (+ 998us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:06.836912 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:06.836924 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:06.836972 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:06.837614 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:06.838115 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:06.839149 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:06.839341 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:06.839440 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:06.841565 (+ 2125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:06.841567 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:06.841567 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:26:06.841583 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:06.842912 (+ 1329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:06.843322 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:06.843323 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:06.843323 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:06.843379 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:06.843683 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:06.843684 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:06.843684 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:06.843806 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:06.843883 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:06.844304 (+ 421us) server_negotiation.cc:299] Negotiation successful
0620 09:26:06.844402 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 09:26:06.844722 17589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:06.835590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:36723 (local address 127.16.43.131:34549)
0620 09:26:06.835787 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:06.835824 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:06.836787 (+ 963us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:06.837043 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:06.837052 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:06.837311 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:06.837498 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:06.837502 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:06.838205 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:06.838207 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:06.839031 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:06.839034 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:06.839445 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:06.839447 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:06.839618 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:06.840115 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:26:06.840123 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:26:06.841456 (+ 1333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:26:06.842999 (+ 1543us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:06.843005 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:06.843007 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:06.843230 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:06.843443 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:06.843444 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:06.843445 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:06.843611 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:06.843872 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:26:06.843879 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:26:06.844044 (+ 165us) client_negotiation.cc:769] Sending connection context
0620 09:26:06.844167 (+ 123us) client_negotiation.cc:240] Negotiation successful
0620 09:26:06.844406 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":95,"threads_started":1}
I20260620 09:26:06.845134 17587 heartbeater.cc:347] Connected to a master server at 127.16.43.190:36723
I20260620 09:26:06.845261 17587 heartbeater.cc:464] Registering TS with master...
I20260620 09:26:06.845428 17587 heartbeater.cc:511] Master 127.16.43.190:36723 requested a full tablet report, sending...
I20260620 09:26:06.845758 17128 ts_manager.cc:194] Registered new tserver with Master: 5c1546eafa874d78aff0d958f467b5cb (127.16.43.131:37139)
I20260620 09:26:06.846124 17128 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.43.131@KRBTEST.COM'} at 127.16.43.131:34549
I20260620 09:26:06.856405 16558 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 09:26:06.863969 17170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:06.858081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50100 (local address 127.16.43.190:36723)
0620 09:26:06.858140 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:06.858142 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:06.858216 (+ 74us) server_negotiation.cc:407] Connection header received
0620 09:26:06.858254 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:06.858255 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:06.858263 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:06.858312 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:06.858840 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:06.859313 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:06.860058 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:06.860250 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:06.860304 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:06.861417 (+ 1113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:06.861418 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:06.861419 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:26:06.861424 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:06.862832 (+ 1408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:06.863177 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:06.863178 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:06.863178 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:06.863244 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:06.863442 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:06.863443 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:06.863443 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:06.863605 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:06.863695 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:06.863844 (+ 149us) server_negotiation.cc:299] Negotiation successful
0620 09:26:06.863902 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 09:26:06.866336 17128 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50100:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 09:26:06.868044 17128 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 09:26:06.875041 17602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:06.870920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38766 (local address 127.16.43.131:37139)
0620 09:26:06.871184 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:06.871208 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:06.871440 (+ 232us) server_negotiation.cc:407] Connection header received
0620 09:26:06.871580 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:06.871582 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:06.871600 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:06.871681 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:26:06.871915 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:06.872915 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:06.874036 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:06.874553 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:06.874667 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:06.874733 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:06.874904 (+ 171us) server_negotiation.cc:299] Negotiation successful
0620 09:26:06.874946 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":90,"threads_started":1}
I20260620 09:26:06.875032 17599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:06.870679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.131:37139 (local address 127.0.0.1:38766)
0620 09:26:06.871314 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:06.871324 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:06.871377 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:06.871776 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:06.871777 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:06.871779 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:26:06.871825 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:06.871827 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:06.873029 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:06.873030 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:06.873937 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:06.873940 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:06.874673 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:06.874674 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:06.874761 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:06.874777 (+ 16us) client_negotiation.cc:769] Sending connection context
0620 09:26:06.874887 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 09:26:06.874946 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614,"thread_start_us":104,"threads_started":1}
I20260620 09:26:06.875257 17601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:06.870775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39576 (local address 127.16.43.129:45599)
0620 09:26:06.871143 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:06.871148 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:06.871162 (+ 14us) server_negotiation.cc:407] Connection header received
0620 09:26:06.871209 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:06.871211 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:06.871229 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:06.871341 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:26:06.871495 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:06.872662 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:06.873948 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:06.874674 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:06.874745 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:06.874800 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:06.875102 (+ 302us) server_negotiation.cc:299] Negotiation successful
0620 09:26:06.875150 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":104,"threads_started":1}
I20260620 09:26:06.875391 17598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:06.870632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.129:45599 (local address 127.0.0.1:39576)
0620 09:26:06.870806 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:06.870852 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:06.870919 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:06.871328 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:06.871331 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:06.871335 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:26:06.871419 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:06.871421 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:06.872775 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:06.872776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:06.873841 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:06.873844 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:06.874838 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:06.874840 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:06.874914 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:06.874920 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 09:26:06.874972 (+ 52us) client_negotiation.cc:240] Negotiation successful
0620 09:26:06.875020 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":66,"threads_started":1}
I20260620 09:26:06.875519 17600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:06.870676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.130:45699 (local address 127.0.0.1:56100)
0620 09:26:06.871124 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:06.871136 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:06.871211 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:06.871478 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:06.871479 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:06.871481 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:26:06.871544 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:06.871546 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:06.872538 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:06.872539 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:06.873860 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:06.873863 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:06.874840 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:06.874841 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:06.874914 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:06.874921 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 09:26:06.874972 (+ 51us) client_negotiation.cc:240] Negotiation successful
0620 09:26:06.875020 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":426,"mutex_wait_us":9,"thread_start_us":97,"threads_started":1}
I20260620 09:26:06.875983 17603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:06.870940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56100 (local address 127.16.43.130:45699)
0620 09:26:06.871105 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:06.871109 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:06.871220 (+ 111us) server_negotiation.cc:407] Connection header received
0620 09:26:06.871292 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:06.871294 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:06.871312 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:06.871388 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:26:06.871626 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:06.872422 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:06.873947 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:06.874674 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:06.874745 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:06.874800 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:06.875064 (+ 264us) server_negotiation.cc:299] Negotiation successful
0620 09:26:06.875885 (+ 821us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":73,"threads_started":1}
I20260620 09:26:06.875965 17521 tablet_service.cc:1467] Processing CreateTablet for tablet 84598b8802bf4fcd88847f1f1bf3449f (DEFAULT_TABLE table=test-table [id=b47979a10b234845bbfbda2030815cfb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:26:06.876163 17253 tablet_service.cc:1467] Processing CreateTablet for tablet 84598b8802bf4fcd88847f1f1bf3449f (DEFAULT_TABLE table=test-table [id=b47979a10b234845bbfbda2030815cfb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:26:06.876412 17521 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84598b8802bf4fcd88847f1f1bf3449f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:26:06.876492 17253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84598b8802bf4fcd88847f1f1bf3449f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:26:06.876577 17387 tablet_service.cc:1467] Processing CreateTablet for tablet 84598b8802bf4fcd88847f1f1bf3449f (DEFAULT_TABLE table=test-table [id=b47979a10b234845bbfbda2030815cfb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:26:06.876857 17387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84598b8802bf4fcd88847f1f1bf3449f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:26:06.879772 17605 tablet_bootstrap.cc:492] T 84598b8802bf4fcd88847f1f1bf3449f P b40e28324fef4972b986d2af3583add8: Bootstrap starting.
I20260620 09:26:06.879999 17606 tablet_bootstrap.cc:492] T 84598b8802bf4fcd88847f1f1bf3449f P fa50fddf5dd14777ac8bc22dae2c058c: Bootstrap starting.
I20260620 09:26:06.880124 17604 tablet_bootstrap.cc:492] T 84598b8802bf4fcd88847f1f1bf3449f P 5c1546eafa874d78aff0d958f467b5cb: Bootstrap starting.
I20260620 09:26:06.880507 17605 tablet_bootstrap.cc:654] T 84598b8802bf4fcd88847f1f1bf3449f P b40e28324fef4972b986d2af3583add8: Neither blocks nor log segments found. Creating new log.
I20260620 09:26:06.880671 17604 tablet_bootstrap.cc:654] T 84598b8802bf4fcd88847f1f1bf3449f P 5c1546eafa874d78aff0d958f467b5cb: Neither blocks nor log segments found. Creating new log.
I20260620 09:26:06.880666 17606 tablet_bootstrap.cc:654] T 84598b8802bf4fcd88847f1f1bf3449f P fa50fddf5dd14777ac8bc22dae2c058c: Neither blocks nor log segments found. Creating new log.
I20260620 09:26:06.881081 17605 log.cc:824] T 84598b8802bf4fcd88847f1f1bf3449f P b40e28324fef4972b986d2af3583add8: Log is configured to *not* fsync() on all Append() calls
I20260620 09:26:06.881268 17604 log.cc:824] T 84598b8802bf4fcd88847f1f1bf3449f P 5c1546eafa874d78aff0d958f467b5cb: Log is configured to *not* fsync() on all Append() calls
I20260620 09:26:06.881268 17606 log.cc:824] T 84598b8802bf4fcd88847f1f1bf3449f P fa50fddf5dd14777ac8bc22dae2c058c: Log is configured to *not* fsync() on all Append() calls
I20260620 09:26:06.882062 17604 tablet_bootstrap.cc:492] T 84598b8802bf4fcd88847f1f1bf3449f P 5c1546eafa874d78aff0d958f467b5cb: No bootstrap required, opened a new log
I20260620 09:26:06.882056 17605 tablet_bootstrap.cc:492] T 84598b8802bf4fcd88847f1f1bf3449f P b40e28324fef4972b986d2af3583add8: No bootstrap required, opened a new log
I20260620 09:26:06.882135 17605 ts_tablet_manager.cc:1397] T 84598b8802bf4fcd88847f1f1bf3449f P b40e28324fef4972b986d2af3583add8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 09:26:06.882135 17604 ts_tablet_manager.cc:1397] T 84598b8802bf4fcd88847f1f1bf3449f P 5c1546eafa874d78aff0d958f467b5cb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 09:26:06.884150 17605 raft_consensus.cc:348] T 84598b8802bf4fcd88847f1f1bf3449f P b40e28324fef4972b986d2af3583add8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1546eafa874d78aff0d958f467b5cb" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 37139 } } peers { permanent_uuid: "b40e28324fef4972b986d2af3583add8" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 45599 } } peers { permanent_uuid: "fa50fddf5dd14777ac8bc22dae2c058c" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 45699 } }
I20260620 09:26:06.884361 17605 raft_consensus.cc:374] T 84598b8802bf4fcd88847f1f1bf3449f P b40e28324fef4972b986d2af3583add8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:26:06.884270 17604 raft_consensus.cc:348] T 84598b8802bf4fcd88847f1f1bf3449f P 5c1546eafa874d78aff0d958f467b5cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1546eafa874d78aff0d958f467b5cb" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 37139 } } peers { permanent_uuid: "b40e28324fef4972b986d2af3583add8" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 45599 } } peers { permanent_uuid: "fa50fddf5dd14777ac8bc22dae2c058c" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 45699 } }
I20260620 09:26:06.884414 17605 raft_consensus.cc:729] T 84598b8802bf4fcd88847f1f1bf3449f P b40e28324fef4972b986d2af3583add8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b40e28324fef4972b986d2af3583add8, State: Initialized, Role: FOLLOWER
I20260620 09:26:06.884455 17604 raft_consensus.cc:374] T 84598b8802bf4fcd88847f1f1bf3449f P 5c1546eafa874d78aff0d958f467b5cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:26:06.884493 17604 raft_consensus.cc:729] T 84598b8802bf4fcd88847f1f1bf3449f P 5c1546eafa874d78aff0d958f467b5cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1546eafa874d78aff0d958f467b5cb, State: Initialized, Role: FOLLOWER
I20260620 09:26:06.884544 17605 consensus_queue.cc:260] T 84598b8802bf4fcd88847f1f1bf3449f P b40e28324fef4972b986d2af3583add8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1546eafa874d78aff0d958f467b5cb" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 37139 } } peers { permanent_uuid: "b40e28324fef4972b986d2af3583add8" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 45599 } } peers { permanent_uuid: "fa50fddf5dd14777ac8bc22dae2c058c" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 45699 } }
I20260620 09:26:06.884606 17604 consensus_queue.cc:260] T 84598b8802bf4fcd88847f1f1bf3449f P 5c1546eafa874d78aff0d958f467b5cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1546eafa874d78aff0d958f467b5cb" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 37139 } } peers { permanent_uuid: "b40e28324fef4972b986d2af3583add8" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 45599 } } peers { permanent_uuid: "fa50fddf5dd14777ac8bc22dae2c058c" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 45699 } }
I20260620 09:26:06.884853 17587 heartbeater.cc:503] Master 127.16.43.190:36723 was elected leader, sending a full tablet report...
I20260620 09:26:06.884840 17319 heartbeater.cc:503] Master 127.16.43.190:36723 was elected leader, sending a full tablet report...
I20260620 09:26:06.884840 17605 ts_tablet_manager.cc:1428] T 84598b8802bf4fcd88847f1f1bf3449f P b40e28324fef4972b986d2af3583add8: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 09:26:06.884840 17604 ts_tablet_manager.cc:1428] T 84598b8802bf4fcd88847f1f1bf3449f P 5c1546eafa874d78aff0d958f467b5cb: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 09:26:06.885352 17606 tablet_bootstrap.cc:492] T 84598b8802bf4fcd88847f1f1bf3449f P fa50fddf5dd14777ac8bc22dae2c058c: No bootstrap required, opened a new log
I20260620 09:26:06.885419 17606 ts_tablet_manager.cc:1397] T 84598b8802bf4fcd88847f1f1bf3449f P fa50fddf5dd14777ac8bc22dae2c058c: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260620 09:26:06.887120 17606 raft_consensus.cc:348] T 84598b8802bf4fcd88847f1f1bf3449f P fa50fddf5dd14777ac8bc22dae2c058c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1546eafa874d78aff0d958f467b5cb" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 37139 } } peers { permanent_uuid: "b40e28324fef4972b986d2af3583add8" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 45599 } } peers { permanent_uuid: "fa50fddf5dd14777ac8bc22dae2c058c" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 45699 } }
I20260620 09:26:06.887275 17606 raft_consensus.cc:374] T 84598b8802bf4fcd88847f1f1bf3449f P fa50fddf5dd14777ac8bc22dae2c058c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:26:06.887297 17606 raft_consensus.cc:729] T 84598b8802bf4fcd88847f1f1bf3449f P fa50fddf5dd14777ac8bc22dae2c058c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa50fddf5dd14777ac8bc22dae2c058c, State: Initialized, Role: FOLLOWER
I20260620 09:26:06.887406 17606 consensus_queue.cc:260] T 84598b8802bf4fcd88847f1f1bf3449f P fa50fddf5dd14777ac8bc22dae2c058c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1546eafa874d78aff0d958f467b5cb" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 37139 } } peers { permanent_uuid: "b40e28324fef4972b986d2af3583add8" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 45599 } } peers { permanent_uuid: "fa50fddf5dd14777ac8bc22dae2c058c" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 45699 } }
I20260620 09:26:06.887696 17606 ts_tablet_manager.cc:1428] T 84598b8802bf4fcd88847f1f1bf3449f P fa50fddf5dd14777ac8bc22dae2c058c: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260620 09:26:06.887732 17453 heartbeater.cc:503] Master 127.16.43.190:36723 was elected leader, sending a full tablet report...
W20260620 09:26:06.915397 17454 tablet.cc:2369] T 84598b8802bf4fcd88847f1f1bf3449f P fa50fddf5dd14777ac8bc22dae2c058c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 09:26:07.003047 17320 tablet.cc:2369] T 84598b8802bf4fcd88847f1f1bf3449f P b40e28324fef4972b986d2af3583add8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 09:26:07.085382 17588 tablet.cc:2369] T 84598b8802bf4fcd88847f1f1bf3449f P 5c1546eafa874d78aff0d958f467b5cb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 09:26:07.278800 17610 raft_consensus.cc:482] T 84598b8802bf4fcd88847f1f1bf3449f P b40e28324fef4972b986d2af3583add8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:26:07.278931 17610 raft_consensus.cc:504] T 84598b8802bf4fcd88847f1f1bf3449f P b40e28324fef4972b986d2af3583add8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1546eafa874d78aff0d958f467b5cb" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 37139 } } peers { permanent_uuid: "b40e28324fef4972b986d2af3583add8" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 45599 } } peers { permanent_uuid: "fa50fddf5dd14777ac8bc22dae2c058c" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 45699 } }
I20260620 09:26:07.279362 17610 leader_election.cc:288] T 84598b8802bf4fcd88847f1f1bf3449f P b40e28324fef4972b986d2af3583add8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1546eafa874d78aff0d958f467b5cb (127.16.43.131:37139), fa50fddf5dd14777ac8bc22dae2c058c (127.16.43.130:45699)
I20260620 09:26:07.283077 17613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:07.279613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.131:37139 (local address 127.16.43.129:44405)
0620 09:26:07.279773 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:07.279786 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:07.279874 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:07.280038 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:07.280039 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:07.280041 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:26:07.280140 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:07.280142 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:07.281114 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:07.281115 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:07.282109 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:07.282112 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:07.282845 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:07.282846 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:07.282900 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:07.282903 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 09:26:07.282953 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 09:26:07.282984 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"mutex_wait_us":72,"thread_start_us":127,"threads_started":1}
I20260620 09:26:07.283102 17602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:07.279832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.129:44405 (local address 127.16.43.131:37139)
0620 09:26:07.279903 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:07.279905 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:07.279916 (+ 11us) server_negotiation.cc:407] Connection header received
0620 09:26:07.279946 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:07.279947 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:07.279963 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:07.280034 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:26:07.280259 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:07.281025 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:07.282215 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:07.282759 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:07.282837 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:07.282846 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:07.282982 (+ 136us) server_negotiation.cc:299] Negotiation successful
0620 09:26:07.283024 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 09:26:07.283372 17614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:07.279613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.130:45699 (local address 127.16.43.129:48427)
0620 09:26:07.279789 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:07.279801 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:07.279883 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:07.280050 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:07.280051 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:07.280052 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:26:07.280124 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:07.280126 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:07.281109 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:07.281110 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:07.282349 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:07.282352 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:07.283104 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:07.283105 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:07.283182 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:07.283186 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 09:26:07.283268 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 09:26:07.283303 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"mutex_wait_us":48,"thread_start_us":71,"threads_started":1}
I20260620 09:26:07.283394 17603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:07.279732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.129:48427 (local address 127.16.43.130:45699)
0620 09:26:07.279796 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:07.279798 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:07.279879 (+ 81us) server_negotiation.cc:407] Connection header received
0620 09:26:07.279946 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:07.279947 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:07.279958 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:07.280034 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:26:07.280282 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:07.281000 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:07.282445 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:07.283013 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:07.283082 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:07.283090 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:07.283281 (+ 191us) server_negotiation.cc:299] Negotiation successful
0620 09:26:07.283327 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 09:26:07.283389 17541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84598b8802bf4fcd88847f1f1bf3449f" candidate_uuid: "b40e28324fef4972b986d2af3583add8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1546eafa874d78aff0d958f467b5cb" is_pre_election: true
I20260620 09:26:07.283581 17541 raft_consensus.cc:2454] T 84598b8802bf4fcd88847f1f1bf3449f P 5c1546eafa874d78aff0d958f467b5cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b40e28324fef4972b986d2af3583add8 in term 0.
I20260620 09:26:07.283743 17407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84598b8802bf4fcd88847f1f1bf3449f" candidate_uuid: "b40e28324fef4972b986d2af3583add8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa50fddf5dd14777ac8bc22dae2c058c" is_pre_election: true
I20260620 09:26:07.283815 17207 leader_election.cc:302] T 84598b8802bf4fcd88847f1f1bf3449f P b40e28324fef4972b986d2af3583add8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c1546eafa874d78aff0d958f467b5cb, b40e28324fef4972b986d2af3583add8; no voters:
I20260620 09:26:07.283905 17407 raft_consensus.cc:2454] T 84598b8802bf4fcd88847f1f1bf3449f P fa50fddf5dd14777ac8bc22dae2c058c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b40e28324fef4972b986d2af3583add8 in term 0.
I20260620 09:26:07.283934 17610 raft_consensus.cc:2781] T 84598b8802bf4fcd88847f1f1bf3449f P b40e28324fef4972b986d2af3583add8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 09:26:07.283980 17610 raft_consensus.cc:482] T 84598b8802bf4fcd88847f1f1bf3449f P b40e28324fef4972b986d2af3583add8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 09:26:07.284008 17610 raft_consensus.cc:3037] T 84598b8802bf4fcd88847f1f1bf3449f P b40e28324fef4972b986d2af3583add8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:26:07.284700 17610 raft_consensus.cc:504] T 84598b8802bf4fcd88847f1f1bf3449f P b40e28324fef4972b986d2af3583add8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1546eafa874d78aff0d958f467b5cb" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 37139 } } peers { permanent_uuid: "b40e28324fef4972b986d2af3583add8" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 45599 } } peers { permanent_uuid: "fa50fddf5dd14777ac8bc22dae2c058c" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 45699 } }
I20260620 09:26:07.284862 17610 leader_election.cc:288] T 84598b8802bf4fcd88847f1f1bf3449f P b40e28324fef4972b986d2af3583add8 [CANDIDATE]: Term 1 election: Requested vote from peers 5c1546eafa874d78aff0d958f467b5cb (127.16.43.131:37139), fa50fddf5dd14777ac8bc22dae2c058c (127.16.43.130:45699)
I20260620 09:26:07.285039 17541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84598b8802bf4fcd88847f1f1bf3449f" candidate_uuid: "b40e28324fef4972b986d2af3583add8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1546eafa874d78aff0d958f467b5cb"
I20260620 09:26:07.285121 17541 raft_consensus.cc:3037] T 84598b8802bf4fcd88847f1f1bf3449f P 5c1546eafa874d78aff0d958f467b5cb [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:26:07.285123 17407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84598b8802bf4fcd88847f1f1bf3449f" candidate_uuid: "b40e28324fef4972b986d2af3583add8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa50fddf5dd14777ac8bc22dae2c058c"
I20260620 09:26:07.285260 17407 raft_consensus.cc:3037] T 84598b8802bf4fcd88847f1f1bf3449f P fa50fddf5dd14777ac8bc22dae2c058c [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:26:07.285854 17541 raft_consensus.cc:2454] T 84598b8802bf4fcd88847f1f1bf3449f P 5c1546eafa874d78aff0d958f467b5cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b40e28324fef4972b986d2af3583add8 in term 1.
I20260620 09:26:07.286027 17407 raft_consensus.cc:2454] T 84598b8802bf4fcd88847f1f1bf3449f P fa50fddf5dd14777ac8bc22dae2c058c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b40e28324fef4972b986d2af3583add8 in term 1.
I20260620 09:26:07.286016 17207 leader_election.cc:302] T 84598b8802bf4fcd88847f1f1bf3449f P b40e28324fef4972b986d2af3583add8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c1546eafa874d78aff0d958f467b5cb, b40e28324fef4972b986d2af3583add8; no voters:
I20260620 09:26:07.286127 17610 raft_consensus.cc:2781] T 84598b8802bf4fcd88847f1f1bf3449f P b40e28324fef4972b986d2af3583add8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:26:07.286285 17610 raft_consensus.cc:686] T 84598b8802bf4fcd88847f1f1bf3449f P b40e28324fef4972b986d2af3583add8 [term 1 LEADER]: Becoming Leader. State: Replica: b40e28324fef4972b986d2af3583add8, State: Running, Role: LEADER
I20260620 09:26:07.286389 17610 consensus_queue.cc:237] T 84598b8802bf4fcd88847f1f1bf3449f P b40e28324fef4972b986d2af3583add8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1546eafa874d78aff0d958f467b5cb" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 37139 } } peers { permanent_uuid: "b40e28324fef4972b986d2af3583add8" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 45599 } } peers { permanent_uuid: "fa50fddf5dd14777ac8bc22dae2c058c" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 45699 } }
I20260620 09:26:07.287217 17128 catalog_manager.cc:5495] T 84598b8802bf4fcd88847f1f1bf3449f P b40e28324fef4972b986d2af3583add8 reported cstate change: term changed from 0 to 1, leader changed from <none> to b40e28324fef4972b986d2af3583add8 (127.16.43.129). New cstate: current_term: 1 leader_uuid: "b40e28324fef4972b986d2af3583add8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1546eafa874d78aff0d958f467b5cb" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 37139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b40e28324fef4972b986d2af3583add8" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 45599 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fa50fddf5dd14777ac8bc22dae2c058c" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 45699 } health_report { overall_health: UNKNOWN } } }
Jun 20 09:26:07 dist-test-slave-706q krb5kdc[17083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947566, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.43.129@KRBTEST.COM
I20260620 09:26:07.325004 17601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:07.316983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39580 (local address 127.16.43.129:45599)
0620 09:26:07.317058 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:07.317060 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:07.317186 (+ 126us) server_negotiation.cc:407] Connection header received
0620 09:26:07.317268 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:07.317269 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:07.317286 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:07.317351 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:07.317982 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:07.318476 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:07.319233 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:07.319418 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:07.319502 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:07.321948 (+ 2446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:07.321960 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:07.321961 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:26:07.321969 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:07.323848 (+ 1879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:07.324183 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:07.324186 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:07.324186 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:07.324255 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:07.324436 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:07.324436 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:07.324436 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:07.324571 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:07.324684 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:07.324818 (+ 134us) server_negotiation.cc:299] Negotiation successful
0620 09:26:07.324880 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
Jun 20 09:26:07 dist-test-slave-706q krb5kdc[17083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947567, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 09:26:07 dist-test-slave-706q krb5kdc[17083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947567, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.43.129@KRBTEST.COM
I20260620 09:26:07.347388 17601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:07.340619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39592 (local address 127.16.43.129:45599)
0620 09:26:07.340739 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:07.340741 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:07.340755 (+ 14us) server_negotiation.cc:407] Connection header received
0620 09:26:07.340785 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:07.340786 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:07.340794 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:07.340847 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:07.341378 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:07.341991 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:07.342688 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:07.342919 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:07.342973 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:07.345252 (+ 2279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:07.345253 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:07.345253 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:26:07.345260 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:07.346408 (+ 1148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:07.346709 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:07.346709 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:07.346710 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:07.346760 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:07.346927 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:07.346928 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:07.346928 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:07.347012 (+ 84us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:07.347101 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:07.347213 (+ 112us) server_negotiation.cc:299] Negotiation successful
0620 09:26:07.347272 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
Jun 20 09:26:07 dist-test-slave-706q krb5kdc[17083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947567, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 09:26:07 dist-test-slave-706q krb5kdc[17083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947567, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.43.129@KRBTEST.COM
I20260620 09:26:07.369871 17601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:07.362809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39598 (local address 127.16.43.129:45599)
0620 09:26:07.362894 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:07.362896 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:07.362939 (+ 43us) server_negotiation.cc:407] Connection header received
0620 09:26:07.362981 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:07.362981 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:07.362990 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:07.363041 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:07.363624 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:07.364125 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:07.364894 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:07.365118 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:07.365188 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:07.367305 (+ 2117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:07.367306 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:07.367306 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:26:07.367313 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:07.368723 (+ 1410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:07.369059 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:07.369060 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:07.369060 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:07.369127 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:07.369299 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:07.369300 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:07.369300 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:07.369410 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:07.369507 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:07.369588 (+ 81us) server_negotiation.cc:299] Negotiation successful
0620 09:26:07.369665 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260620 09:26:07.370414 17233 tablet_service.cc:2908] Rejecting scan request for tablet 84598b8802bf4fcd88847f1f1bf3449f: Uninitialized: clean time has not yet been initialized
I20260620 09:26:07.371259 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 17190
I20260620 09:26:07.376716 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 17324
I20260620 09:26:07.381824 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 17458
I20260620 09:26:07.386709 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 17096
2026-06-20T09:26:07Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2466 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781947562949448-16558-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:26:07 dist-test-slave-706q krb5kdc[17629](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:26:07 dist-test-slave-706q krb5kdc[17629](info): set up 2 sockets
Jun 20 09:26:07 dist-test-slave-706q krb5kdc[17629](info): commencing operation
krb5kdc: starting...
W20260620 09:26:08.448159 16558 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781947562949448-16558-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781947562949448-16558-0/krb5kdc/test-user.keytab.
Jun 20 09:26:08 dist-test-slave-706q krb5kdc[17629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947568, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:26:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:26:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.43.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.43.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.43.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.190.keytab.
Entry for principal kudu/127.16.43.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.190.keytab.
I20260620 09:26:08.568363 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781947562949448-16558-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781947562949448-16558-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781947562949448-16558-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.43.190:42073
--webserver_interface=127.16.43.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.190.keytab
--principal=kudu/127.16.43.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.43.148:39715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.43.190:42073
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781947562949448-16558-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781947562949448-16558-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781947562949448-16558-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:26:08.649587 17642 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:26:08.649863 17642 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:26:08.649900 17642 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:26:08.651062 17642 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:26:08.651141 17642 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 09:26:08.651162 17642 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:26:08.651182 17642 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:26:08.651206 17642 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:26:08.651234 17642 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:26:08.652730 17642 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:39715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781947562949448-16558-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781947562949448-16558-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.43.190:42073
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.190.keytab
--principal=kudu/127.16.43.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.43.190:42073
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.43.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17642
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781947562949448-16558-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:26:08.653043 17642 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:26:08.653285 17642 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:26:08.658301 17650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:26:08.658315 17647 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:26:08.658301 17648 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:26:08.658551 17642 server_base.cc:1034] running on GCE node
I20260620 09:26:08.658998 17642 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:26:08.659305 17642 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:26:08.660476 17642 hybrid_clock.cc:648] HybridClock initialized: now 1781947568660440 us; error 54 us; skew 500 ppm
Jun 20 09:26:08 dist-test-slave-706q krb5kdc[17629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947568, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:26:08.663498 17642 init.cc:377] Logged in from keytab as kudu/127.16.43.190@KRBTEST.COM (short username kudu)
I20260620 09:26:08.664176 17642 webserver.cc:460] Webserver started at http://127.16.43.190:39983/ using document root <none> and password file <none>
I20260620 09:26:08.664496 17642 fs_manager.cc:359] Metadata directory not provided
I20260620 09:26:08.664593 17642 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:26:08.664752 17642 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:26:08.665997 17642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781947562949448-16558-0/minicluster-data/master-0/data/instance:
uuid: "2325df10f30d4fafbf003522017927f9"
format_stamp: "Formatted at 2026-06-20 09:26:08 on dist-test-slave-706q"
server_key: "9606dc5fcd38c24b5d91a0031d8b84b1"
server_key_iv: "f9e1c27b910b229eaea884e82f8aa3ca"
server_key_version: "encryptionkey@0"
I20260620 09:26:08.666430 17642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781947562949448-16558-0/minicluster-data/master-0/wal/instance:
uuid: "2325df10f30d4fafbf003522017927f9"
format_stamp: "Formatted at 2026-06-20 09:26:08 on dist-test-slave-706q"
server_key: "9606dc5fcd38c24b5d91a0031d8b84b1"
server_key_iv: "f9e1c27b910b229eaea884e82f8aa3ca"
server_key_version: "encryptionkey@0"
I20260620 09:26:08.668085 17642 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.004s sys 0.000s
I20260620 09:26:08.669090 17657 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:08.669317 17642 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:26:08.669435 17642 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781947562949448-16558-0/minicluster-data/master-0/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781947562949448-16558-0/minicluster-data/master-0/wal
uuid: "2325df10f30d4fafbf003522017927f9"
format_stamp: "Formatted at 2026-06-20 09:26:08 on dist-test-slave-706q"
server_key: "9606dc5fcd38c24b5d91a0031d8b84b1"
server_key_iv: "f9e1c27b910b229eaea884e82f8aa3ca"
server_key_version: "encryptionkey@0"
I20260620 09:26:08.669530 17642 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781947562949448-16558-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781947562949448-16558-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781947562949448-16558-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:26:08.695312 17642 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:26:08.697775 17642 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:26:08.697973 17642 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:26:08.703297 17642 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.190:42073
I20260620 09:26:08.703357 17719 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.190:42073 every 8 connection(s)
I20260620 09:26:08.703738 17642 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
I20260620 09:26:08.704066 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 17642
I20260620 09:26:08.704182 16558 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781947562949448-16558-0/minicluster-data/master-0/wal/instance
I20260620 09:26:08.704375 16558 external_mini_cluster.cc:1428] Setting key bc2cf675e712e86177bb8a2937a1ae9b
I20260620 09:26:08.704589 17720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:26:08.707685 17720 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2325df10f30d4fafbf003522017927f9: Bootstrap starting.
I20260620 09:26:08.708323 17720 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2325df10f30d4fafbf003522017927f9: Neither blocks nor log segments found. Creating new log.
I20260620 09:26:08.708624 17720 log.cc:824] T 00000000000000000000000000000000 P 2325df10f30d4fafbf003522017927f9: Log is configured to *not* fsync() on all Append() calls
I20260620 09:26:08.709440 17720 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2325df10f30d4fafbf003522017927f9: No bootstrap required, opened a new log
Jun 20 09:26:08 dist-test-slave-706q krb5kdc[17629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947568, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.43.190@KRBTEST.COM
I20260620 09:26:08.711550 17720 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2325df10f30d4fafbf003522017927f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2325df10f30d4fafbf003522017927f9" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 42073 } }
I20260620 09:26:08.711752 17720 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2325df10f30d4fafbf003522017927f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:26:08.711815 17720 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2325df10f30d4fafbf003522017927f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2325df10f30d4fafbf003522017927f9, State: Initialized, Role: FOLLOWER
I20260620 09:26:08.711953 17720 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2325df10f30d4fafbf003522017927f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2325df10f30d4fafbf003522017927f9" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 42073 } }
I20260620 09:26:08.712071 17720 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2325df10f30d4fafbf003522017927f9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:26:08.712111 17720 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2325df10f30d4fafbf003522017927f9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:26:08.712172 17720 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2325df10f30d4fafbf003522017927f9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:26:08.712924 17720 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2325df10f30d4fafbf003522017927f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2325df10f30d4fafbf003522017927f9" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 42073 } }
I20260620 09:26:08.713032 17720 leader_election.cc:302] T 00000000000000000000000000000000 P 2325df10f30d4fafbf003522017927f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2325df10f30d4fafbf003522017927f9; no voters:
I20260620 09:26:08.713200 17720 leader_election.cc:288] T 00000000000000000000000000000000 P 2325df10f30d4fafbf003522017927f9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:26:08.713259 17725 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2325df10f30d4fafbf003522017927f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:26:08.713464 17720 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2325df10f30d4fafbf003522017927f9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:26:08.713470 17725 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2325df10f30d4fafbf003522017927f9 [term 1 LEADER]: Becoming Leader. State: Replica: 2325df10f30d4fafbf003522017927f9, State: Running, Role: LEADER
I20260620 09:26:08.713675 17725 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2325df10f30d4fafbf003522017927f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2325df10f30d4fafbf003522017927f9" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 42073 } }
I20260620 09:26:08.714052 17726 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2325df10f30d4fafbf003522017927f9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2325df10f30d4fafbf003522017927f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2325df10f30d4fafbf003522017927f9" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 42073 } } }
I20260620 09:26:08.714140 17726 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2325df10f30d4fafbf003522017927f9 [sys.catalog]: This master's current role is: LEADER
I20260620 09:26:08.714596 17727 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2325df10f30d4fafbf003522017927f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2325df10f30d4fafbf003522017927f9. Latest consensus state: current_term: 1 leader_uuid: "2325df10f30d4fafbf003522017927f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2325df10f30d4fafbf003522017927f9" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 42073 } } }
I20260620 09:26:08.714743 17727 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2325df10f30d4fafbf003522017927f9 [sys.catalog]: This master's current role is: LEADER
I20260620 09:26:08.714885 17734 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:26:08.715574 17734 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:26:08.715817 17723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:08.705008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51250 (local address 127.16.43.190:42073)
0620 09:26:08.705215 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:08.705223 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:08.705249 (+ 26us) server_negotiation.cc:407] Connection header received
0620 09:26:08.705716 (+ 467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:08.705722 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:08.705780 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:08.705932 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:08.706572 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:08.707505 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:08.708419 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:08.708923 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:08.709041 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:08.711402 (+ 2361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:08.711408 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:08.711409 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:26:08.711422 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:08.713557 (+ 2135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:08.714004 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:08.714007 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:08.714007 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:08.714130 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:08.714396 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:08.714396 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:08.714397 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:08.714738 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:08.714905 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:08.715388 (+ 483us) server_negotiation.cc:299] Negotiation successful
0620 09:26:08.715560 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":100,"threads_started":1}
I20260620 09:26:08.717157 17734 catalog_manager.cc:1347] Generated new cluster ID: 561668c1472d4d19b22b18e7674d8ee0
I20260620 09:26:08.717209 17734 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:26:08.739609 17734 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:26:08.740242 17734 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:26:08.750142 17734 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2325df10f30d4fafbf003522017927f9: Generated new TSK 0
I20260620 09:26:08.750380 17734 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.43.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.43.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.43.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.129.keytab.
Entry for principal kudu/127.16.43.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.129.keytab.
I20260620 09:26:08.782372 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781947562949448-16558-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781947562949448-16558-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781947562949448-16558-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781947562949448-16558-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.43.129:0
--local_ip_for_outbound_sockets=127.16.43.129
--webserver_interface=127.16.43.129
--webserver_port=0
--tserver_master_addrs=127.16.43.190:42073
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.129.keytab
--principal=kudu/127.16.43.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.43.148:39715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781947562949448-16558-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781947562949448-16558-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781947562949448-16558-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:26:08.866274 17746 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:26:08.866501 17746 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:26:08.866526 17746 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:26:08.867777 17746 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:26:08.867823 17746 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:26:08.867846 17746 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 09:26:08.867869 17746 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.43.129
I20260620 09:26:08.869318 17746 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:39715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781947562949448-16558-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781947562949448-16558-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.129.keytab
--principal=kudu/127.16.43.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.43.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781947562949448-16558-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.43.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.43.190:42073
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17746
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.43.129
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781947562949448-16558-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:26:08.869554 17746 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:26:08.869812 17746 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:26:08.874987 17754 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:26:08.874962 17751 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:26:08.875016 17752 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:26:08.875380 17746 server_base.cc:1034] running on GCE node
I20260620 09:26:08.875674 17746 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:26:08.875931 17746 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:26:08.877074 17746 hybrid_clock.cc:648] HybridClock initialized: now 1781947568877058 us; error 45 us; skew 500 ppm
Jun 20 09:26:08 dist-test-slave-706q krb5kdc[17629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947568, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:26:08.879633 17746 init.cc:377] Logged in from keytab as kudu/127.16.43.129@KRBTEST.COM (short username kudu)
I20260620 09:26:08.880098 17746 webserver.cc:460] Webserver started at http://127.16.43.129:35883/ using document root <none> and password file <none>
I20260620 09:26:08.880332 17746 fs_manager.cc:359] Metadata directory not provided
I20260620 09:26:08.880409 17746 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:26:08.880559 17746 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:26:08.881668 17746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781947562949448-16558-0/minicluster-data/ts-0/data/instance:
uuid: "afc6a0bc3a714e809d81ab77676481a5"
format_stamp: "Formatted at 2026-06-20 09:26:08 on dist-test-slave-706q"
server_key: "8ba1f870629e2729b85bcddeb18ba619"
server_key_iv: "4cfbc5b21cb5fc97cabe7b1fb103105e"
server_key_version: "encryptionkey@0"
I20260620 09:26:08.882035 17746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781947562949448-16558-0/minicluster-data/ts-0/wal/instance:
uuid: "afc6a0bc3a714e809d81ab77676481a5"
format_stamp: "Formatted at 2026-06-20 09:26:08 on dist-test-slave-706q"
server_key: "8ba1f870629e2729b85bcddeb18ba619"
server_key_iv: "4cfbc5b21cb5fc97cabe7b1fb103105e"
server_key_version: "encryptionkey@0"
I20260620 09:26:08.883369 17746 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 09:26:08.884244 17761 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:08.884466 17746 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 09:26:08.884564 17746 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781947562949448-16558-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781947562949448-16558-0/minicluster-data/ts-0/wal
uuid: "afc6a0bc3a714e809d81ab77676481a5"
format_stamp: "Formatted at 2026-06-20 09:26:08 on dist-test-slave-706q"
server_key: "8ba1f870629e2729b85bcddeb18ba619"
server_key_iv: "4cfbc5b21cb5fc97cabe7b1fb103105e"
server_key_version: "encryptionkey@0"
I20260620 09:26:08.884647 17746 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781947562949448-16558-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781947562949448-16558-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781947562949448-16558-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:26:08.896893 17746 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:26:08.899283 17746 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:26:08.899456 17746 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:26:08.900113 17746 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:26:08.900166 17746 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:08.900202 17746 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:26:08.900223 17746 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 09:26:08 dist-test-slave-706q krb5kdc[17629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947568, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.129@KRBTEST.COM for kudu/127.16.43.190@KRBTEST.COM
I20260620 09:26:08.907627 17746 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.129:46835
I20260620 09:26:08.908011 17746 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781947562949448-16558-0/minicluster-data/ts-0/data/info.pb
I20260620 09:26:08.907975 17876 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.129:46835 every 8 connection(s)
I20260620 09:26:08.910315 17723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:08.900570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.129:57147 (local address 127.16.43.190:42073)
0620 09:26:08.900643 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:08.900646 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:08.900892 (+ 246us) server_negotiation.cc:407] Connection header received
0620 09:26:08.901359 (+ 467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:08.901360 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:08.901380 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:08.901463 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:08.902102 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:08.902979 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:08.904133 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:08.904520 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:08.904602 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:08.907287 (+ 2685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:08.907289 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:08.907290 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:26:08.907298 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:08.908997 (+ 1699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:08.909352 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:08.909352 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:08.909353 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:08.909417 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:08.909640 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:08.909642 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:08.909642 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:08.909772 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:08.909838 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:08.910124 (+ 286us) server_negotiation.cc:299] Negotiation successful
0620 09:26:08.910202 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 09:26:08.910331 17770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:08.900559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:42073 (local address 127.16.43.129:57147)
0620 09:26:08.900734 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:08.900775 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:08.901267 (+ 492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:08.901560 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:08.901563 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:08.901790 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:08.901991 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:08.901995 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:08.903098 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:08.903099 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:08.904025 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:08.904028 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:08.904631 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:08.904632 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:08.904766 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:08.905337 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:26:08.905355 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:26:08.907153 (+ 1798us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:26:08.909067 (+ 1914us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:08.909072 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:08.909073 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:08.909284 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:08.909468 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:08.909468 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:08.909469 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:08.909555 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:08.909848 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:26:08.909851 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:26:08.909962 (+ 111us) client_negotiation.cc:769] Sending connection context
0620 09:26:08.910048 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 09:26:08.910166 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":107,"threads_started":1}
I20260620 09:26:08.910900 17877 heartbeater.cc:347] Connected to a master server at 127.16.43.190:42073
I20260620 09:26:08.911000 17877 heartbeater.cc:464] Registering TS with master...
I20260620 09:26:08.911183 17877 heartbeater.cc:511] Master 127.16.43.190:42073 requested a full tablet report, sending...
I20260620 09:26:08.911793 17673 ts_manager.cc:194] Registered new tserver with Master: afc6a0bc3a714e809d81ab77676481a5 (127.16.43.129:46835)
I20260620 09:26:08.912734 17673 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.43.129@KRBTEST.COM'} at 127.16.43.129:57147
I20260620 09:26:08.917610 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 17746
I20260620 09:26:08.917819 16558 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781947562949448-16558-0/minicluster-data/ts-0/wal/instance
I20260620 09:26:08.918049 16558 external_mini_cluster.cc:1428] Setting key a18bd25a48b40d039271e7f49ba18c33
I20260620 09:26:08.918445 17723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:08.911990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.129:53307 (local address 127.16.43.190:42073)
0620 09:26:08.912032 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:08.912034 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:08.912135 (+ 101us) server_negotiation.cc:407] Connection header received
0620 09:26:08.912204 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:08.912204 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:08.912212 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:08.912260 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:08.912622 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:08.913290 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:08.914129 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:08.914322 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:08.914377 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:08.915522 (+ 1145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:08.915524 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:08.915524 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:26:08.915575 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:08.916982 (+ 1407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:08.917403 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:08.917404 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:08.917405 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:08.917461 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:08.917779 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:08.917780 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:08.917781 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:08.917964 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:08.918020 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:08.918276 (+ 256us) server_negotiation.cc:299] Negotiation successful
0620 09:26:08.918342 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 09:26:08.918555 17883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:08.911949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:42073 (local address 127.16.43.129:53307)
0620 09:26:08.912078 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:08.912090 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:08.912149 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:08.912273 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:08.912273 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:08.912473 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:08.912538 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:08.912539 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:08.913388 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:08.913389 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:08.913983 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:08.913985 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:08.914412 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:08.914413 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:08.914478 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:08.915060 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:26:08.915065 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:26:08.915418 (+ 353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:26:08.917079 (+ 1661us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:08.917080 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:08.917080 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:08.917282 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:08.917543 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:08.917544 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:08.917544 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:08.917626 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:08.918020 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:26:08.918021 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:26:08.918080 (+ 59us) client_negotiation.cc:769] Sending connection context
0620 09:26:08.918173 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 09:26:08.918444 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":72,"threads_started":1}
WARNING: no policy specified for kudu/127.16.43.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.43.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.43.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.130.keytab.
Entry for principal kudu/127.16.43.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.130.keytab.
I20260620 09:26:08.946766 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781947562949448-16558-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781947562949448-16558-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781947562949448-16558-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781947562949448-16558-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.43.130:0
--local_ip_for_outbound_sockets=127.16.43.130
--webserver_interface=127.16.43.130
--webserver_port=0
--tserver_master_addrs=127.16.43.190:42073
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.130.keytab
--principal=kudu/127.16.43.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.43.148:39715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781947562949448-16558-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781947562949448-16558-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781947562949448-16558-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:26:09.029311 17886 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:26:09.029561 17886 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:26:09.029604 17886 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:26:09.030779 17886 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:26:09.030848 17886 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:26:09.030884 17886 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 09:26:09.030942 17886 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.43.130
I20260620 09:26:09.032459 17886 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:39715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781947562949448-16558-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781947562949448-16558-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.130.keytab
--principal=kudu/127.16.43.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.43.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781947562949448-16558-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.43.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.43.190:42073
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17886
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.43.130
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781947562949448-16558-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:26:09.032743 17886 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:26:09.032989 17886 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:26:09.038069 17891 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:26:09.038113 17894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:26:09.038137 17892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:26:09.038335 17886 server_base.cc:1034] running on GCE node
I20260620 09:26:09.038530 17886 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:26:09.038787 17886 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:26:09.039952 17886 hybrid_clock.cc:648] HybridClock initialized: now 1781947569039927 us; error 40 us; skew 500 ppm
Jun 20 09:26:09 dist-test-slave-706q krb5kdc[17629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947569, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:26:09.042448 17886 init.cc:377] Logged in from keytab as kudu/127.16.43.130@KRBTEST.COM (short username kudu)
I20260620 09:26:09.042935 17886 webserver.cc:460] Webserver started at http://127.16.43.130:37975/ using document root <none> and password file <none>
I20260620 09:26:09.043169 17886 fs_manager.cc:359] Metadata directory not provided
I20260620 09:26:09.043236 17886 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:26:09.043390 17886 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:26:09.044615 17886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781947562949448-16558-0/minicluster-data/ts-1/data/instance:
uuid: "7c582790769f481cbba1421f2cb1dc5e"
format_stamp: "Formatted at 2026-06-20 09:26:09 on dist-test-slave-706q"
server_key: "3b992fd9e01b617c9b9ca44149eeacaa"
server_key_iv: "f754c15bc7b0832b7236677fbbec7b3f"
server_key_version: "encryptionkey@0"
I20260620 09:26:09.044986 17886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781947562949448-16558-0/minicluster-data/ts-1/wal/instance:
uuid: "7c582790769f481cbba1421f2cb1dc5e"
format_stamp: "Formatted at 2026-06-20 09:26:09 on dist-test-slave-706q"
server_key: "3b992fd9e01b617c9b9ca44149eeacaa"
server_key_iv: "f754c15bc7b0832b7236677fbbec7b3f"
server_key_version: "encryptionkey@0"
I20260620 09:26:09.046375 17886 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 09:26:09.047166 17901 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:09.047353 17886 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 09:26:09.047451 17886 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781947562949448-16558-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781947562949448-16558-0/minicluster-data/ts-1/wal
uuid: "7c582790769f481cbba1421f2cb1dc5e"
format_stamp: "Formatted at 2026-06-20 09:26:09 on dist-test-slave-706q"
server_key: "3b992fd9e01b617c9b9ca44149eeacaa"
server_key_iv: "f754c15bc7b0832b7236677fbbec7b3f"
server_key_version: "encryptionkey@0"
I20260620 09:26:09.047562 17886 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781947562949448-16558-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781947562949448-16558-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781947562949448-16558-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:26:09.054734 17886 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:26:09.056587 17886 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:26:09.056747 17886 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:26:09.057341 17886 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:26:09.057392 17886 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:09.057425 17886 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:26:09.057442 17886 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 09:26:09 dist-test-slave-706q krb5kdc[17629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947569, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.130@KRBTEST.COM for kudu/127.16.43.190@KRBTEST.COM
I20260620 09:26:09.064762 17886 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.130:43425
I20260620 09:26:09.064858 18016 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.130:43425 every 8 connection(s)
I20260620 09:26:09.065187 17886 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781947562949448-16558-0/minicluster-data/ts-1/data/info.pb
I20260620 09:26:09.067420 17723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:09.057747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.130:39047 (local address 127.16.43.190:42073)
0620 09:26:09.057822 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:09.057824 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:09.058091 (+ 267us) server_negotiation.cc:407] Connection header received
0620 09:26:09.058634 (+ 543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:09.058636 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:09.058648 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:09.058782 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:09.059764 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:09.060318 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:09.061514 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:09.061792 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:09.061882 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:09.064735 (+ 2853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:09.064737 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:09.064737 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:26:09.064746 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:09.066170 (+ 1424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:09.066543 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:09.066544 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:09.066544 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:09.066593 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:09.066832 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:09.066833 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:09.066834 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:09.066965 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:09.067038 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:09.067263 (+ 225us) server_negotiation.cc:299] Negotiation successful
0620 09:26:09.067344 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 09:26:09.067520 17910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:09.057749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:42073 (local address 127.16.43.130:39047)
0620 09:26:09.057942 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:09.057984 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:09.058522 (+ 538us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:09.058867 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:09.058872 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:09.059308 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:09.059647 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:09.059653 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:09.060442 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:09.060443 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:09.061385 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:09.061387 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:09.061910 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:09.061912 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:09.062047 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:09.062625 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:26:09.062637 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:26:09.064612 (+ 1975us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:26:09.066244 (+ 1632us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:09.066249 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:09.066251 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:09.066475 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:09.066649 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:09.066650 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:09.066650 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:09.066743 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:09.067043 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:26:09.067048 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:26:09.067159 (+ 111us) client_negotiation.cc:769] Sending connection context
0620 09:26:09.067241 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 09:26:09.067371 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":124,"threads_started":1}
I20260620 09:26:09.068094 18017 heartbeater.cc:347] Connected to a master server at 127.16.43.190:42073
I20260620 09:26:09.068176 18017 heartbeater.cc:464] Registering TS with master...
I20260620 09:26:09.068357 18017 heartbeater.cc:511] Master 127.16.43.190:42073 requested a full tablet report, sending...
I20260620 09:26:09.068735 17674 ts_manager.cc:194] Registered new tserver with Master: 7c582790769f481cbba1421f2cb1dc5e (127.16.43.130:43425)
I20260620 09:26:09.069133 17674 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.43.130@KRBTEST.COM'} at 127.16.43.130:39047
I20260620 09:26:09.071867 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 17886
I20260620 09:26:09.071970 16558 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781947562949448-16558-0/minicluster-data/ts-1/wal/instance
I20260620 09:26:09.072157 16558 external_mini_cluster.cc:1428] Setting key 11b305f3ca314b56b1b68e6b63c48680
I20260620 09:26:09.075284 17723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:09.069205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.130:56287 (local address 127.16.43.190:42073)
0620 09:26:09.069348 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:09.069350 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:09.069359 (+ 9us) server_negotiation.cc:407] Connection header received
0620 09:26:09.069381 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:09.069381 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:09.069393 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:09.069463 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:09.069867 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:09.070398 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:09.071065 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:09.071244 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:09.071350 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:09.072350 (+ 1000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:09.072351 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:09.072352 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:26:09.072359 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:09.074125 (+ 1766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:09.074573 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:09.074574 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:09.074574 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:09.074623 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:09.074820 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:09.074820 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:09.074820 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:09.074948 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:09.074999 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:09.075156 (+ 157us) server_negotiation.cc:299] Negotiation successful
0620 09:26:09.075210 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121}
I20260620 09:26:09.075376 18023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:09.069147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:42073 (local address 127.16.43.130:56287)
0620 09:26:09.069263 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:09.069273 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:09.069328 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:09.069479 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:09.069480 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:09.069687 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:09.069759 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:09.069761 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:09.070490 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:09.070491 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:09.070981 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:09.070982 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:09.071353 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:09.071354 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:09.071428 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:09.071855 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:26:09.071861 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:26:09.072259 (+ 398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:26:09.074250 (+ 1991us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:09.074251 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:09.074251 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:09.074494 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:09.074688 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:09.074689 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:09.074689 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:09.074759 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:09.075010 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:26:09.075010 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:26:09.075061 (+ 51us) client_negotiation.cc:769] Sending connection context
0620 09:26:09.075142 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 09:26:09.075256 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":62,"threads_started":1}
WARNING: no policy specified for kudu/127.16.43.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.43.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.43.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.131.keytab.
Entry for principal kudu/127.16.43.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.131.keytab.
I20260620 09:26:09.099159 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781947562949448-16558-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781947562949448-16558-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781947562949448-16558-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781947562949448-16558-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.43.131:0
--local_ip_for_outbound_sockets=127.16.43.131
--webserver_interface=127.16.43.131
--webserver_port=0
--tserver_master_addrs=127.16.43.190:42073
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.131.keytab
--principal=kudu/127.16.43.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.43.148:39715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781947562949448-16558-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781947562949448-16558-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781947562949448-16558-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:26:09.186120 18026 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:26:09.186363 18026 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:26:09.186403 18026 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:26:09.187700 18026 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:26:09.187760 18026 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:26:09.187794 18026 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 09:26:09.187832 18026 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.43.131
I20260620 09:26:09.189330 18026 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:39715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781947562949448-16558-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781947562949448-16558-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.131.keytab
--principal=kudu/127.16.43.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.43.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781947562949448-16558-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.43.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.43.190:42073
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18026
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.43.131
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781947562949448-16558-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:26:09.189613 18026 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:26:09.189852 18026 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:26:09.194761 18031 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:26:09.194854 18032 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:26:09.194932 18034 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:26:09.195132 18026 server_base.cc:1034] running on GCE node
I20260620 09:26:09.195336 18026 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:26:09.195582 18026 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:26:09.196760 18026 hybrid_clock.cc:648] HybridClock initialized: now 1781947569196744 us; error 32 us; skew 500 ppm
Jun 20 09:26:09 dist-test-slave-706q krb5kdc[17629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947569, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:26:09.199220 18026 init.cc:377] Logged in from keytab as kudu/127.16.43.131@KRBTEST.COM (short username kudu)
I20260620 09:26:09.199750 18026 webserver.cc:460] Webserver started at http://127.16.43.131:45599/ using document root <none> and password file <none>
I20260620 09:26:09.199988 18026 fs_manager.cc:359] Metadata directory not provided
I20260620 09:26:09.200032 18026 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:26:09.200165 18026 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:26:09.201226 18026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781947562949448-16558-0/minicluster-data/ts-2/data/instance:
uuid: "33d1881b38b9401b85c7fb2a3d976d24"
format_stamp: "Formatted at 2026-06-20 09:26:09 on dist-test-slave-706q"
server_key: "20bf3bca77ad6968188fcb61f6ddb961"
server_key_iv: "a7f541ee71cc40a8dd6efb8a3f3e5f13"
server_key_version: "encryptionkey@0"
I20260620 09:26:09.201642 18026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781947562949448-16558-0/minicluster-data/ts-2/wal/instance:
uuid: "33d1881b38b9401b85c7fb2a3d976d24"
format_stamp: "Formatted at 2026-06-20 09:26:09 on dist-test-slave-706q"
server_key: "20bf3bca77ad6968188fcb61f6ddb961"
server_key_iv: "a7f541ee71cc40a8dd6efb8a3f3e5f13"
server_key_version: "encryptionkey@0"
I20260620 09:26:09.203079 18026 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 09:26:09.203971 18041 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:09.204223 18026 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 09:26:09.204309 18026 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781947562949448-16558-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781947562949448-16558-0/minicluster-data/ts-2/wal
uuid: "33d1881b38b9401b85c7fb2a3d976d24"
format_stamp: "Formatted at 2026-06-20 09:26:09 on dist-test-slave-706q"
server_key: "20bf3bca77ad6968188fcb61f6ddb961"
server_key_iv: "a7f541ee71cc40a8dd6efb8a3f3e5f13"
server_key_version: "encryptionkey@0"
I20260620 09:26:09.204365 18026 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781947562949448-16558-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781947562949448-16558-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781947562949448-16558-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:26:09.217048 18026 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:26:09.219399 18026 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:26:09.219516 18026 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:26:09.220067 18026 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:26:09.220106 18026 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:09.220130 18026 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:26:09.220180 18026 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:09.227255 18026 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.131:33169
I20260620 09:26:09.227385 18156 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.131:33169 every 8 connection(s)
I20260620 09:26:09.227696 18026 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781947562949448-16558-0/minicluster-data/ts-2/data/info.pb
Jun 20 09:26:09 dist-test-slave-706q krb5kdc[17629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947569, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.131@KRBTEST.COM for kudu/127.16.43.190@KRBTEST.COM
I20260620 09:26:09.231263 17723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:09.220609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.131:47061 (local address 127.16.43.190:42073)
0620 09:26:09.220655 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:09.220662 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:09.220944 (+ 282us) server_negotiation.cc:407] Connection header received
0620 09:26:09.221490 (+ 546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:09.221492 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:09.221505 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:09.221564 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:09.222584 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:09.223414 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:09.225187 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:09.225404 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:09.225466 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:09.228359 (+ 2893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:09.228361 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:09.228361 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:26:09.228369 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:09.229832 (+ 1463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:09.230259 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:09.230260 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:09.230261 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:09.230322 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:09.230597 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:09.230598 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:09.230598 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:09.230718 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:09.230798 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:09.231074 (+ 276us) server_negotiation.cc:299] Negotiation successful
0620 09:26:09.231156 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 09:26:09.231467 18050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:09.220609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:42073 (local address 127.16.43.131:47061)
0620 09:26:09.220814 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:09.220850 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:09.221370 (+ 520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:09.221690 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:09.221707 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:09.222120 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:09.222446 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:09.222454 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:09.223523 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:09.223524 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:09.225086 (+ 1562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:09.225089 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:09.225487 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:09.225488 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:09.225654 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:09.226364 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:26:09.226376 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:26:09.228248 (+ 1872us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:26:09.229937 (+ 1689us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:09.229944 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:09.229946 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:09.230176 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:09.230400 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:09.230401 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:09.230402 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:09.230519 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:09.230789 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:26:09.230795 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:26:09.230943 (+ 148us) client_negotiation.cc:769] Sending connection context
0620 09:26:09.231055 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 09:26:09.231218 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":106,"threads_started":1}
I20260620 09:26:09.232191 18157 heartbeater.cc:347] Connected to a master server at 127.16.43.190:42073
I20260620 09:26:09.232278 18157 heartbeater.cc:464] Registering TS with master...
I20260620 09:26:09.232470 18157 heartbeater.cc:511] Master 127.16.43.190:42073 requested a full tablet report, sending...
I20260620 09:26:09.232928 17674 ts_manager.cc:194] Registered new tserver with Master: 33d1881b38b9401b85c7fb2a3d976d24 (127.16.43.131:33169)
I20260620 09:26:09.233403 17674 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.43.131@KRBTEST.COM'} at 127.16.43.131:47061
I20260620 09:26:09.234603 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 18026
I20260620 09:26:09.234704 16558 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781947562949448-16558-0/minicluster-data/ts-2/wal/instance
I20260620 09:26:09.234889 16558 external_mini_cluster.cc:1428] Setting key 0a9511e05d87434232a5e14bdcf7934b
I20260620 09:26:09.235823 16558 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 09:26:09.239876 17723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:09.233393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.131:37221 (local address 127.16.43.190:42073)
0620 09:26:09.233433 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:09.233435 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:09.233601 (+ 166us) server_negotiation.cc:407] Connection header received
0620 09:26:09.233619 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:09.233619 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:09.233627 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:09.233664 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:09.234036 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:09.234662 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:09.235786 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:09.236056 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:09.236121 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:09.237277 (+ 1156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:09.237280 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:09.237280 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:26:09.237288 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:09.238689 (+ 1401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:09.239063 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:09.239064 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:09.239064 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:09.239116 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:09.239325 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:09.239326 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:09.239326 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:09.239440 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:09.239516 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:09.239702 (+ 186us) server_negotiation.cc:299] Negotiation successful
0620 09:26:09.239777 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 09:26:09.239887 18163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:09.233305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:42073 (local address 127.16.43.131:37221)
0620 09:26:09.233448 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:09.233462 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:09.233534 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:09.233673 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:09.233674 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:09.233891 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:09.233950 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:09.233952 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:09.234831 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:09.234832 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:09.235664 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:09.235666 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:09.236174 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:09.236175 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:09.236267 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:09.236738 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:26:09.236743 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:26:09.237164 (+ 421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:26:09.238799 (+ 1635us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:09.238800 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:09.238800 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:09.238974 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:09.239191 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:09.239192 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:09.239193 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:09.239252 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:09.239507 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:26:09.239508 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:26:09.239583 (+ 75us) client_negotiation.cc:769] Sending connection context
0620 09:26:09.239674 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 09:26:09.239779 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":80,"threads_started":1}
Jun 20 09:26:09 dist-test-slave-706q krb5kdc[17629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947569, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 09:26:09 dist-test-slave-706q krb5kdc[17629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947569, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.43.190@KRBTEST.COM
I20260620 09:26:09.260021 17723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:09.252664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51264 (local address 127.16.43.190:42073)
0620 09:26:09.252739 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:09.252741 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:09.252757 (+ 16us) server_negotiation.cc:407] Connection header received
0620 09:26:09.252780 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:09.252780 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:09.252789 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:09.252856 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:09.253427 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:09.253921 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:09.254699 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:09.254891 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:09.254979 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:09.257346 (+ 2367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:09.257348 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:09.257348 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:26:09.257358 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:09.258730 (+ 1372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:09.259098 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:09.259099 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:09.259100 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:09.259158 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:09.259359 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:09.259360 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:09.259360 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:09.259476 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:09.259618 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:09.259729 (+ 111us) server_negotiation.cc:299] Negotiation successful
0620 09:26:09.259910 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 09:26:09.261859 17674 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51264:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 09:26:09.263500 17674 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 09:26:09.270478 18178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:09.266105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.129:46835 (local address 127.0.0.1:56372)
0620 09:26:09.266466 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:09.266476 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:09.266541 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:09.266729 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:09.266733 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:09.266735 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:26:09.266837 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:09.266840 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:09.268082 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:09.268083 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:09.269414 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:09.269417 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:09.270161 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:09.270162 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:09.270240 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:09.270253 (+ 13us) client_negotiation.cc:769] Sending connection context
0620 09:26:09.270324 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 09:26:09.270385 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"mutex_wait_us":59,"thread_start_us":83,"threads_started":1}
I20260620 09:26:09.270502 18177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:09.266229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56372 (local address 127.16.43.129:46835)
0620 09:26:09.266411 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:09.266417 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:09.266534 (+ 117us) server_negotiation.cc:407] Connection header received
0620 09:26:09.266609 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:09.266610 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:09.266624 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:09.266697 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:26:09.266940 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:09.267969 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:09.269506 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:09.270067 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:09.270125 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:09.270208 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:09.270364 (+ 156us) server_negotiation.cc:299] Negotiation successful
0620 09:26:09.270406 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":99,"threads_started":1}
I20260620 09:26:09.270519 18179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:09.266500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41734 (local address 127.16.43.131:33169)
0620 09:26:09.266966 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:09.266970 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:09.266979 (+ 9us) server_negotiation.cc:407] Connection header received
0620 09:26:09.267014 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:09.267017 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:09.267031 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:09.267114 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:26:09.267236 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:09.268354 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:09.269605 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:09.270072 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:09.270165 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:09.270223 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:09.270378 (+ 155us) server_negotiation.cc:299] Negotiation successful
0620 09:26:09.270424 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":78,"threads_started":1}
I20260620 09:26:09.271005 18175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:09.266295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.131:33169 (local address 127.0.0.1:41734)
0620 09:26:09.266660 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:09.266687 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:09.266729 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:09.267126 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:09.267127 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:09.267129 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:26:09.267170 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:09.267171 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:09.268432 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:09.268433 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:09.269533 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:09.269535 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:09.270179 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:09.270180 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:09.270250 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:09.270254 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 09:26:09.270330 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 09:26:09.270905 (+ 575us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"mutex_wait_us":534,"thread_start_us":75,"threads_started":1}
I20260620 09:26:09.271315 18174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:09.266071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.130:43425 (local address 127.0.0.1:41884)
0620 09:26:09.266257 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:09.266314 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:09.266383 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:09.266766 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:09.266767 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:09.266769 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:26:09.266837 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:09.266839 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:09.268237 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:09.268238 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:09.269428 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:09.269430 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:09.271069 (+ 1639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:09.271070 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:09.271134 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:09.271137 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 09:26:09.271216 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 09:26:09.271248 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":81,"threads_started":1}
I20260620 09:26:09.271895 18176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:09.266172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41884 (local address 127.16.43.130:43425)
0620 09:26:09.266455 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:09.266460 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:09.266470 (+ 10us) server_negotiation.cc:407] Connection header received
0620 09:26:09.266506 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:09.266508 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:09.266524 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:09.266601 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:26:09.267013 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:09.268160 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:09.270242 (+ 2082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:09.270890 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:09.270998 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:09.271064 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:09.271746 (+ 682us) server_negotiation.cc:299] Negotiation successful
0620 09:26:09.271792 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":83,"threads_started":1}
I20260620 09:26:09.272140 18091 tablet_service.cc:1467] Processing CreateTablet for tablet e4e6cc7d88e54932bdd5fd56d1373794 (DEFAULT_TABLE table=test-table [id=bd4927c8842d4ae8ab4698c9579ca367]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:26:09.272166 17811 tablet_service.cc:1467] Processing CreateTablet for tablet e4e6cc7d88e54932bdd5fd56d1373794 (DEFAULT_TABLE table=test-table [id=bd4927c8842d4ae8ab4698c9579ca367]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:26:09.272464 18091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4e6cc7d88e54932bdd5fd56d1373794. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:26:09.272506 17811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4e6cc7d88e54932bdd5fd56d1373794. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:26:09.272467 17951 tablet_service.cc:1467] Processing CreateTablet for tablet e4e6cc7d88e54932bdd5fd56d1373794 (DEFAULT_TABLE table=test-table [id=bd4927c8842d4ae8ab4698c9579ca367]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:26:09.272768 17951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4e6cc7d88e54932bdd5fd56d1373794. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:26:09.275043 18180 tablet_bootstrap.cc:492] T e4e6cc7d88e54932bdd5fd56d1373794 P 33d1881b38b9401b85c7fb2a3d976d24: Bootstrap starting.
I20260620 09:26:09.276036 18180 tablet_bootstrap.cc:654] T e4e6cc7d88e54932bdd5fd56d1373794 P 33d1881b38b9401b85c7fb2a3d976d24: Neither blocks nor log segments found. Creating new log.
I20260620 09:26:09.276335 18181 tablet_bootstrap.cc:492] T e4e6cc7d88e54932bdd5fd56d1373794 P afc6a0bc3a714e809d81ab77676481a5: Bootstrap starting.
I20260620 09:26:09.276417 18180 log.cc:824] T e4e6cc7d88e54932bdd5fd56d1373794 P 33d1881b38b9401b85c7fb2a3d976d24: Log is configured to *not* fsync() on all Append() calls
I20260620 09:26:09.276712 18182 tablet_bootstrap.cc:492] T e4e6cc7d88e54932bdd5fd56d1373794 P 7c582790769f481cbba1421f2cb1dc5e: Bootstrap starting.
I20260620 09:26:09.276984 18181 tablet_bootstrap.cc:654] T e4e6cc7d88e54932bdd5fd56d1373794 P afc6a0bc3a714e809d81ab77676481a5: Neither blocks nor log segments found. Creating new log.
I20260620 09:26:09.277325 18182 tablet_bootstrap.cc:654] T e4e6cc7d88e54932bdd5fd56d1373794 P 7c582790769f481cbba1421f2cb1dc5e: Neither blocks nor log segments found. Creating new log.
I20260620 09:26:09.277323 18181 log.cc:824] T e4e6cc7d88e54932bdd5fd56d1373794 P afc6a0bc3a714e809d81ab77676481a5: Log is configured to *not* fsync() on all Append() calls
I20260620 09:26:09.277645 18182 log.cc:824] T e4e6cc7d88e54932bdd5fd56d1373794 P 7c582790769f481cbba1421f2cb1dc5e: Log is configured to *not* fsync() on all Append() calls
I20260620 09:26:09.278123 18181 tablet_bootstrap.cc:492] T e4e6cc7d88e54932bdd5fd56d1373794 P afc6a0bc3a714e809d81ab77676481a5: No bootstrap required, opened a new log
I20260620 09:26:09.278194 18181 ts_tablet_manager.cc:1397] T e4e6cc7d88e54932bdd5fd56d1373794 P afc6a0bc3a714e809d81ab77676481a5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 09:26:09.278363 18180 tablet_bootstrap.cc:492] T e4e6cc7d88e54932bdd5fd56d1373794 P 33d1881b38b9401b85c7fb2a3d976d24: No bootstrap required, opened a new log
I20260620 09:26:09.278366 18182 tablet_bootstrap.cc:492] T e4e6cc7d88e54932bdd5fd56d1373794 P 7c582790769f481cbba1421f2cb1dc5e: No bootstrap required, opened a new log
I20260620 09:26:09.278434 18182 ts_tablet_manager.cc:1397] T e4e6cc7d88e54932bdd5fd56d1373794 P 7c582790769f481cbba1421f2cb1dc5e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 09:26:09.278434 18180 ts_tablet_manager.cc:1397] T e4e6cc7d88e54932bdd5fd56d1373794 P 33d1881b38b9401b85c7fb2a3d976d24: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 09:26:09.279935 18181 raft_consensus.cc:348] T e4e6cc7d88e54932bdd5fd56d1373794 P afc6a0bc3a714e809d81ab77676481a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c582790769f481cbba1421f2cb1dc5e" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 43425 } } peers { permanent_uuid: "afc6a0bc3a714e809d81ab77676481a5" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 46835 } } peers { permanent_uuid: "33d1881b38b9401b85c7fb2a3d976d24" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 33169 } }
I20260620 09:26:09.280093 18181 raft_consensus.cc:374] T e4e6cc7d88e54932bdd5fd56d1373794 P afc6a0bc3a714e809d81ab77676481a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:26:09.280138 18181 raft_consensus.cc:729] T e4e6cc7d88e54932bdd5fd56d1373794 P afc6a0bc3a714e809d81ab77676481a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: afc6a0bc3a714e809d81ab77676481a5, State: Initialized, Role: FOLLOWER
I20260620 09:26:09.280243 18181 consensus_queue.cc:260] T e4e6cc7d88e54932bdd5fd56d1373794 P afc6a0bc3a714e809d81ab77676481a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c582790769f481cbba1421f2cb1dc5e" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 43425 } } peers { permanent_uuid: "afc6a0bc3a714e809d81ab77676481a5" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 46835 } } peers { permanent_uuid: "33d1881b38b9401b85c7fb2a3d976d24" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 33169 } }
I20260620 09:26:09.280521 18181 ts_tablet_manager.cc:1428] T e4e6cc7d88e54932bdd5fd56d1373794 P afc6a0bc3a714e809d81ab77676481a5: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 09:26:09.280457 18180 raft_consensus.cc:348] T e4e6cc7d88e54932bdd5fd56d1373794 P 33d1881b38b9401b85c7fb2a3d976d24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c582790769f481cbba1421f2cb1dc5e" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 43425 } } peers { permanent_uuid: "afc6a0bc3a714e809d81ab77676481a5" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 46835 } } peers { permanent_uuid: "33d1881b38b9401b85c7fb2a3d976d24" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 33169 } }
I20260620 09:26:09.280458 18182 raft_consensus.cc:348] T e4e6cc7d88e54932bdd5fd56d1373794 P 7c582790769f481cbba1421f2cb1dc5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c582790769f481cbba1421f2cb1dc5e" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 43425 } } peers { permanent_uuid: "afc6a0bc3a714e809d81ab77676481a5" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 46835 } } peers { permanent_uuid: "33d1881b38b9401b85c7fb2a3d976d24" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 33169 } }
I20260620 09:26:09.280522 17877 heartbeater.cc:503] Master 127.16.43.190:42073 was elected leader, sending a full tablet report...
I20260620 09:26:09.280606 18180 raft_consensus.cc:374] T e4e6cc7d88e54932bdd5fd56d1373794 P 33d1881b38b9401b85c7fb2a3d976d24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:26:09.280606 18182 raft_consensus.cc:374] T e4e6cc7d88e54932bdd5fd56d1373794 P 7c582790769f481cbba1421f2cb1dc5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:26:09.280642 18180 raft_consensus.cc:729] T e4e6cc7d88e54932bdd5fd56d1373794 P 33d1881b38b9401b85c7fb2a3d976d24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33d1881b38b9401b85c7fb2a3d976d24, State: Initialized, Role: FOLLOWER
I20260620 09:26:09.280642 18182 raft_consensus.cc:729] T e4e6cc7d88e54932bdd5fd56d1373794 P 7c582790769f481cbba1421f2cb1dc5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c582790769f481cbba1421f2cb1dc5e, State: Initialized, Role: FOLLOWER
I20260620 09:26:09.280763 18182 consensus_queue.cc:260] T e4e6cc7d88e54932bdd5fd56d1373794 P 7c582790769f481cbba1421f2cb1dc5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c582790769f481cbba1421f2cb1dc5e" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 43425 } } peers { permanent_uuid: "afc6a0bc3a714e809d81ab77676481a5" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 46835 } } peers { permanent_uuid: "33d1881b38b9401b85c7fb2a3d976d24" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 33169 } }
I20260620 09:26:09.280754 18180 consensus_queue.cc:260] T e4e6cc7d88e54932bdd5fd56d1373794 P 33d1881b38b9401b85c7fb2a3d976d24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c582790769f481cbba1421f2cb1dc5e" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 43425 } } peers { permanent_uuid: "afc6a0bc3a714e809d81ab77676481a5" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 46835 } } peers { permanent_uuid: "33d1881b38b9401b85c7fb2a3d976d24" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 33169 } }
I20260620 09:26:09.281009 18182 ts_tablet_manager.cc:1428] T e4e6cc7d88e54932bdd5fd56d1373794 P 7c582790769f481cbba1421f2cb1dc5e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 09:26:09.281009 18180 ts_tablet_manager.cc:1428] T e4e6cc7d88e54932bdd5fd56d1373794 P 33d1881b38b9401b85c7fb2a3d976d24: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 09:26:09.281124 18157 heartbeater.cc:503] Master 127.16.43.190:42073 was elected leader, sending a full tablet report...
I20260620 09:26:09.281124 18017 heartbeater.cc:503] Master 127.16.43.190:42073 was elected leader, sending a full tablet report...
W20260620 09:26:09.315786 18018 tablet.cc:2369] T e4e6cc7d88e54932bdd5fd56d1373794 P 7c582790769f481cbba1421f2cb1dc5e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 09:26:09.408921 17878 tablet.cc:2369] T e4e6cc7d88e54932bdd5fd56d1373794 P afc6a0bc3a714e809d81ab77676481a5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 09:26:09.478348 18158 tablet.cc:2369] T e4e6cc7d88e54932bdd5fd56d1373794 P 33d1881b38b9401b85c7fb2a3d976d24: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 09:26:09.642745 18186 raft_consensus.cc:482] T e4e6cc7d88e54932bdd5fd56d1373794 P afc6a0bc3a714e809d81ab77676481a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:26:09.642874 18186 raft_consensus.cc:504] T e4e6cc7d88e54932bdd5fd56d1373794 P afc6a0bc3a714e809d81ab77676481a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c582790769f481cbba1421f2cb1dc5e" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 43425 } } peers { permanent_uuid: "afc6a0bc3a714e809d81ab77676481a5" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 46835 } } peers { permanent_uuid: "33d1881b38b9401b85c7fb2a3d976d24" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 33169 } }
I20260620 09:26:09.643302 18186 leader_election.cc:288] T e4e6cc7d88e54932bdd5fd56d1373794 P afc6a0bc3a714e809d81ab77676481a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7c582790769f481cbba1421f2cb1dc5e (127.16.43.130:43425), 33d1881b38b9401b85c7fb2a3d976d24 (127.16.43.131:33169)
I20260620 09:26:09.646437 18179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:09.643696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.129:57917 (local address 127.16.43.131:33169)
0620 09:26:09.643773 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:09.643775 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:09.643786 (+ 11us) server_negotiation.cc:407] Connection header received
0620 09:26:09.643889 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:09.643891 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:09.643902 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:09.643973 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:26:09.644142 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:09.644674 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:09.645720 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:09.646071 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:09.646126 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:09.646133 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:09.646314 (+ 181us) server_negotiation.cc:299] Negotiation successful
0620 09:26:09.646352 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 09:26:09.646452 18176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:09.643637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.129:52043 (local address 127.16.43.130:43425)
0620 09:26:09.643686 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:09.643688 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:09.643827 (+ 139us) server_negotiation.cc:407] Connection header received
0620 09:26:09.643971 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:09.643972 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:09.643982 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:09.644041 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:26:09.644239 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:09.644739 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:09.645724 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:09.646086 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:09.646177 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:09.646185 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:09.646347 (+ 162us) server_negotiation.cc:299] Negotiation successful
0620 09:26:09.646388 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 09:26:09.646471 18189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:09.643557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.131:33169 (local address 127.16.43.129:57917)
0620 09:26:09.643724 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:09.643737 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:09.643800 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:09.643978 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:09.643980 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:09.643982 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:26:09.644058 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:09.644061 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:09.644766 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:09.644767 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:09.645622 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:09.645625 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:09.646160 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:09.646161 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:09.646231 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:09.646237 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 09:26:09.646284 (+ 47us) client_negotiation.cc:240] Negotiation successful
0620 09:26:09.646346 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"mutex_wait_us":17,"thread_start_us":76,"threads_started":1}
I20260620 09:26:09.646479 18190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:09.643559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.130:43425 (local address 127.16.43.129:52043)
0620 09:26:09.643749 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:09.643765 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:09.643850 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:09.644060 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:09.644061 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:09.644062 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:26:09.644151 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:09.644153 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:09.644829 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:09.644830 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:09.645620 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:09.645622 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:09.646190 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:09.646191 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:09.646252 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:09.646255 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 09:26:09.646327 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 09:26:09.646364 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"mutex_wait_us":61,"thread_start_us":77,"threads_started":1}
I20260620 09:26:09.646857 18111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4e6cc7d88e54932bdd5fd56d1373794" candidate_uuid: "afc6a0bc3a714e809d81ab77676481a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33d1881b38b9401b85c7fb2a3d976d24" is_pre_election: true
I20260620 09:26:09.646965 17971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4e6cc7d88e54932bdd5fd56d1373794" candidate_uuid: "afc6a0bc3a714e809d81ab77676481a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c582790769f481cbba1421f2cb1dc5e" is_pre_election: true
I20260620 09:26:09.647037 18111 raft_consensus.cc:2454] T e4e6cc7d88e54932bdd5fd56d1373794 P 33d1881b38b9401b85c7fb2a3d976d24 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate afc6a0bc3a714e809d81ab77676481a5 in term 0.
I20260620 09:26:09.647126 17971 raft_consensus.cc:2454] T e4e6cc7d88e54932bdd5fd56d1373794 P 7c582790769f481cbba1421f2cb1dc5e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate afc6a0bc3a714e809d81ab77676481a5 in term 0.
I20260620 09:26:09.647285 17762 leader_election.cc:302] T e4e6cc7d88e54932bdd5fd56d1373794 P afc6a0bc3a714e809d81ab77676481a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33d1881b38b9401b85c7fb2a3d976d24, afc6a0bc3a714e809d81ab77676481a5; no voters:
I20260620 09:26:09.647460 18186 raft_consensus.cc:2781] T e4e6cc7d88e54932bdd5fd56d1373794 P afc6a0bc3a714e809d81ab77676481a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 09:26:09.647567 18186 raft_consensus.cc:482] T e4e6cc7d88e54932bdd5fd56d1373794 P afc6a0bc3a714e809d81ab77676481a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 09:26:09.647603 18186 raft_consensus.cc:3037] T e4e6cc7d88e54932bdd5fd56d1373794 P afc6a0bc3a714e809d81ab77676481a5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:26:09.648408 18186 raft_consensus.cc:504] T e4e6cc7d88e54932bdd5fd56d1373794 P afc6a0bc3a714e809d81ab77676481a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c582790769f481cbba1421f2cb1dc5e" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 43425 } } peers { permanent_uuid: "afc6a0bc3a714e809d81ab77676481a5" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 46835 } } peers { permanent_uuid: "33d1881b38b9401b85c7fb2a3d976d24" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 33169 } }
I20260620 09:26:09.648566 18186 leader_election.cc:288] T e4e6cc7d88e54932bdd5fd56d1373794 P afc6a0bc3a714e809d81ab77676481a5 [CANDIDATE]: Term 1 election: Requested vote from peers 7c582790769f481cbba1421f2cb1dc5e (127.16.43.130:43425), 33d1881b38b9401b85c7fb2a3d976d24 (127.16.43.131:33169)
I20260620 09:26:09.648703 18111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4e6cc7d88e54932bdd5fd56d1373794" candidate_uuid: "afc6a0bc3a714e809d81ab77676481a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33d1881b38b9401b85c7fb2a3d976d24"
I20260620 09:26:09.648801 18111 raft_consensus.cc:3037] T e4e6cc7d88e54932bdd5fd56d1373794 P 33d1881b38b9401b85c7fb2a3d976d24 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:26:09.648845 17971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4e6cc7d88e54932bdd5fd56d1373794" candidate_uuid: "afc6a0bc3a714e809d81ab77676481a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c582790769f481cbba1421f2cb1dc5e"
I20260620 09:26:09.648948 17971 raft_consensus.cc:3037] T e4e6cc7d88e54932bdd5fd56d1373794 P 7c582790769f481cbba1421f2cb1dc5e [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:26:09.649536 18111 raft_consensus.cc:2454] T e4e6cc7d88e54932bdd5fd56d1373794 P 33d1881b38b9401b85c7fb2a3d976d24 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate afc6a0bc3a714e809d81ab77676481a5 in term 1.
I20260620 09:26:09.649744 17762 leader_election.cc:302] T e4e6cc7d88e54932bdd5fd56d1373794 P afc6a0bc3a714e809d81ab77676481a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33d1881b38b9401b85c7fb2a3d976d24, afc6a0bc3a714e809d81ab77676481a5; no voters:
I20260620 09:26:09.649892 18186 raft_consensus.cc:2781] T e4e6cc7d88e54932bdd5fd56d1373794 P afc6a0bc3a714e809d81ab77676481a5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:26:09.649909 17971 raft_consensus.cc:2454] T e4e6cc7d88e54932bdd5fd56d1373794 P 7c582790769f481cbba1421f2cb1dc5e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate afc6a0bc3a714e809d81ab77676481a5 in term 1.
I20260620 09:26:09.650158 18186 raft_consensus.cc:686] T e4e6cc7d88e54932bdd5fd56d1373794 P afc6a0bc3a714e809d81ab77676481a5 [term 1 LEADER]: Becoming Leader. State: Replica: afc6a0bc3a714e809d81ab77676481a5, State: Running, Role: LEADER
I20260620 09:26:09.650302 18186 consensus_queue.cc:237] T e4e6cc7d88e54932bdd5fd56d1373794 P afc6a0bc3a714e809d81ab77676481a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c582790769f481cbba1421f2cb1dc5e" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 43425 } } peers { permanent_uuid: "afc6a0bc3a714e809d81ab77676481a5" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 46835 } } peers { permanent_uuid: "33d1881b38b9401b85c7fb2a3d976d24" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 33169 } }
I20260620 09:26:09.651170 17673 catalog_manager.cc:5495] T e4e6cc7d88e54932bdd5fd56d1373794 P afc6a0bc3a714e809d81ab77676481a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to afc6a0bc3a714e809d81ab77676481a5 (127.16.43.129). New cstate: current_term: 1 leader_uuid: "afc6a0bc3a714e809d81ab77676481a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c582790769f481cbba1421f2cb1dc5e" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 43425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "afc6a0bc3a714e809d81ab77676481a5" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 46835 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "33d1881b38b9401b85c7fb2a3d976d24" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 33169 } health_report { overall_health: UNKNOWN } } }
I20260620 09:26:09.714316 18177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:09.711566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56382 (local address 127.16.43.129:46835)
0620 09:26:09.711619 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:09.711621 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:09.711631 (+ 10us) server_negotiation.cc:407] Connection header received
0620 09:26:09.711821 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:09.711822 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:09.711842 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:09.711915 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:26:09.712073 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:09.712550 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:09.713367 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:09.713557 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:09.713613 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:09.713823 (+ 210us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:26:09.713917 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:26:09.713980 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:09.714192 (+ 212us) server_negotiation.cc:299] Negotiation successful
0620 09:26:09.714222 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 09:26:09.716439 18111 raft_consensus.cc:1261] T e4e6cc7d88e54932bdd5fd56d1373794 P 33d1881b38b9401b85c7fb2a3d976d24 [term 1 FOLLOWER]: Refusing update from remote peer afc6a0bc3a714e809d81ab77676481a5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:26:09.716769 17971 raft_consensus.cc:1261] T e4e6cc7d88e54932bdd5fd56d1373794 P 7c582790769f481cbba1421f2cb1dc5e [term 1 FOLLOWER]: Refusing update from remote peer afc6a0bc3a714e809d81ab77676481a5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:26:09.716873 18191 consensus_queue.cc:1035] T e4e6cc7d88e54932bdd5fd56d1373794 P afc6a0bc3a714e809d81ab77676481a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33d1881b38b9401b85c7fb2a3d976d24" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 33169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:26:09.717036 18186 consensus_queue.cc:1035] T e4e6cc7d88e54932bdd5fd56d1373794 P afc6a0bc3a714e809d81ab77676481a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c582790769f481cbba1421f2cb1dc5e" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 43425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:26:09.720052 18195 mvcc.cc:205] Tried to move back new op lower bound from 7298857245555109888 to 7298857245288243200. Current Snapshot: MvccSnapshot[applied={T|T < 7298857245555109888}]
I20260620 09:26:09.722867 17674 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51264:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:26:09.722968 17674 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51264:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:26:09.723757 17674 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2325df10f30d4fafbf003522017927f9: Sending DeleteTablet for 3 replicas of tablet e4e6cc7d88e54932bdd5fd56d1373794
I20260620 09:26:09.724097 17951 tablet_service.cc:1514] Processing DeleteTablet for tablet e4e6cc7d88e54932bdd5fd56d1373794 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:26:09 UTC) from {username='kudu', principal='kudu/127.16.43.190@KRBTEST.COM'} at 127.0.0.1:41884
I20260620 09:26:09.724174 17811 tablet_service.cc:1514] Processing DeleteTablet for tablet e4e6cc7d88e54932bdd5fd56d1373794 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:26:09 UTC) from {username='kudu', principal='kudu/127.16.43.190@KRBTEST.COM'} at 127.0.0.1:56372
I20260620 09:26:09.724392 18204 tablet_replica.cc:330] T e4e6cc7d88e54932bdd5fd56d1373794 P 7c582790769f481cbba1421f2cb1dc5e: stopping tablet replica
I20260620 09:26:09.724390 18091 tablet_service.cc:1514] Processing DeleteTablet for tablet e4e6cc7d88e54932bdd5fd56d1373794 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:26:09 UTC) from {username='kudu', principal='kudu/127.16.43.190@KRBTEST.COM'} at 127.0.0.1:41734
I20260620 09:26:09.724421 18205 tablet_replica.cc:330] T e4e6cc7d88e54932bdd5fd56d1373794 P afc6a0bc3a714e809d81ab77676481a5: stopping tablet replica
I20260620 09:26:09.724519 18204 raft_consensus.cc:2229] T e4e6cc7d88e54932bdd5fd56d1373794 P 7c582790769f481cbba1421f2cb1dc5e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:26:09.724627 18204 raft_consensus.cc:2258] T e4e6cc7d88e54932bdd5fd56d1373794 P 7c582790769f481cbba1421f2cb1dc5e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:26:09.724603 17674 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51264:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 09:26:09.724737 17674 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 09:26:09.724650 18205 raft_consensus.cc:2229] T e4e6cc7d88e54932bdd5fd56d1373794 P afc6a0bc3a714e809d81ab77676481a5 [term 1 LEADER]: Raft consensus shutting down.
I20260620 09:26:09.725016 18205 raft_consensus.cc:2258] T e4e6cc7d88e54932bdd5fd56d1373794 P afc6a0bc3a714e809d81ab77676481a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:26:09.725013 18206 tablet_replica.cc:330] T e4e6cc7d88e54932bdd5fd56d1373794 P 33d1881b38b9401b85c7fb2a3d976d24: stopping tablet replica
I20260620 09:26:09.725108 18206 raft_consensus.cc:2229] T e4e6cc7d88e54932bdd5fd56d1373794 P 33d1881b38b9401b85c7fb2a3d976d24 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:26:09.725216 18206 raft_consensus.cc:2258] T e4e6cc7d88e54932bdd5fd56d1373794 P 33d1881b38b9401b85c7fb2a3d976d24 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:26:09.725623 18205 ts_tablet_manager.cc:1905] T e4e6cc7d88e54932bdd5fd56d1373794 P afc6a0bc3a714e809d81ab77676481a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:26:09.725740 18206 ts_tablet_manager.cc:1905] T e4e6cc7d88e54932bdd5fd56d1373794 P 33d1881b38b9401b85c7fb2a3d976d24: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:26:09.726362 17811 tablet_service.cc:1467] Processing CreateTablet for tablet dcb0cb99ed8c44f39b2a1f33929861b0 (DEFAULT_TABLE table=test-table [id=29d93747acdf416cbac91274de79fd99]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:26:09.726490 17811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcb0cb99ed8c44f39b2a1f33929861b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:26:09.727582 18205 ts_tablet_manager.cc:1918] T e4e6cc7d88e54932bdd5fd56d1373794 P afc6a0bc3a714e809d81ab77676481a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:26:09.727628 18091 tablet_service.cc:1467] Processing CreateTablet for tablet dcb0cb99ed8c44f39b2a1f33929861b0 (DEFAULT_TABLE table=test-table [id=29d93747acdf416cbac91274de79fd99]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:26:09.727754 18205 log.cc:1196] T e4e6cc7d88e54932bdd5fd56d1373794 P afc6a0bc3a714e809d81ab77676481a5: Deleting WAL directory at /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781947562949448-16558-0/minicluster-data/ts-0/wal/wals/e4e6cc7d88e54932bdd5fd56d1373794
I20260620 09:26:09.727806 18206 ts_tablet_manager.cc:1918] T e4e6cc7d88e54932bdd5fd56d1373794 P 33d1881b38b9401b85c7fb2a3d976d24: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:26:09.727897 18206 log.cc:1196] T e4e6cc7d88e54932bdd5fd56d1373794 P 33d1881b38b9401b85c7fb2a3d976d24: Deleting WAL directory at /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781947562949448-16558-0/minicluster-data/ts-2/wal/wals/e4e6cc7d88e54932bdd5fd56d1373794
I20260620 09:26:09.728032 17951 tablet_service.cc:1467] Processing CreateTablet for tablet dcb0cb99ed8c44f39b2a1f33929861b0 (DEFAULT_TABLE table=test-table [id=29d93747acdf416cbac91274de79fd99]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:26:09.728142 17951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcb0cb99ed8c44f39b2a1f33929861b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:26:09.728183 18206 ts_tablet_manager.cc:1939] T e4e6cc7d88e54932bdd5fd56d1373794 P 33d1881b38b9401b85c7fb2a3d976d24: Deleting consensus metadata
I20260620 09:26:09.728371 18205 ts_tablet_manager.cc:1939] T e4e6cc7d88e54932bdd5fd56d1373794 P afc6a0bc3a714e809d81ab77676481a5: Deleting consensus metadata
I20260620 09:26:09.728821 17658 catalog_manager.cc:4841] TS afc6a0bc3a714e809d81ab77676481a5 (127.16.43.129:46835): tablet e4e6cc7d88e54932bdd5fd56d1373794 (table test-table [id=bd4927c8842d4ae8ab4698c9579ca367]) successfully deleted
I20260620 09:26:09.729324 18182 tablet_bootstrap.cc:492] T dcb0cb99ed8c44f39b2a1f33929861b0 P 7c582790769f481cbba1421f2cb1dc5e: Bootstrap starting.
I20260620 09:26:09.729439 17658 catalog_manager.cc:4841] TS 33d1881b38b9401b85c7fb2a3d976d24 (127.16.43.131:33169): tablet e4e6cc7d88e54932bdd5fd56d1373794 (table test-table [id=bd4927c8842d4ae8ab4698c9579ca367]) successfully deleted
I20260620 09:26:09.727805 18091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcb0cb99ed8c44f39b2a1f33929861b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:26:09.729730 18182 tablet_bootstrap.cc:654] T dcb0cb99ed8c44f39b2a1f33929861b0 P 7c582790769f481cbba1421f2cb1dc5e: Neither blocks nor log segments found. Creating new log.
I20260620 09:26:09.730379 18182 tablet_bootstrap.cc:492] T dcb0cb99ed8c44f39b2a1f33929861b0 P 7c582790769f481cbba1421f2cb1dc5e: No bootstrap required, opened a new log
I20260620 09:26:09.730443 18182 ts_tablet_manager.cc:1397] T dcb0cb99ed8c44f39b2a1f33929861b0 P 7c582790769f481cbba1421f2cb1dc5e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 09:26:09.730610 18180 tablet_bootstrap.cc:492] T dcb0cb99ed8c44f39b2a1f33929861b0 P 33d1881b38b9401b85c7fb2a3d976d24: Bootstrap starting.
I20260620 09:26:09.730600 18182 raft_consensus.cc:348] T dcb0cb99ed8c44f39b2a1f33929861b0 P 7c582790769f481cbba1421f2cb1dc5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afc6a0bc3a714e809d81ab77676481a5" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 46835 } } peers { permanent_uuid: "7c582790769f481cbba1421f2cb1dc5e" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 43425 } } peers { permanent_uuid: "33d1881b38b9401b85c7fb2a3d976d24" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 33169 } }
I20260620 09:26:09.730667 18182 raft_consensus.cc:374] T dcb0cb99ed8c44f39b2a1f33929861b0 P 7c582790769f481cbba1421f2cb1dc5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:26:09.730688 18182 raft_consensus.cc:729] T dcb0cb99ed8c44f39b2a1f33929861b0 P 7c582790769f481cbba1421f2cb1dc5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c582790769f481cbba1421f2cb1dc5e, State: Initialized, Role: FOLLOWER
I20260620 09:26:09.730736 18182 consensus_queue.cc:260] T dcb0cb99ed8c44f39b2a1f33929861b0 P 7c582790769f481cbba1421f2cb1dc5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afc6a0bc3a714e809d81ab77676481a5" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 46835 } } peers { permanent_uuid: "7c582790769f481cbba1421f2cb1dc5e" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 43425 } } peers { permanent_uuid: "33d1881b38b9401b85c7fb2a3d976d24" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 33169 } }
I20260620 09:26:09.730865 18182 ts_tablet_manager.cc:1428] T dcb0cb99ed8c44f39b2a1f33929861b0 P 7c582790769f481cbba1421f2cb1dc5e: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 09:26:09.731021 18180 tablet_bootstrap.cc:654] T dcb0cb99ed8c44f39b2a1f33929861b0 P 33d1881b38b9401b85c7fb2a3d976d24: Neither blocks nor log segments found. Creating new log.
I20260620 09:26:09.731033 18181 tablet_bootstrap.cc:492] T dcb0cb99ed8c44f39b2a1f33929861b0 P afc6a0bc3a714e809d81ab77676481a5: Bootstrap starting.
I20260620 09:26:09.731448 18181 tablet_bootstrap.cc:654] T dcb0cb99ed8c44f39b2a1f33929861b0 P afc6a0bc3a714e809d81ab77676481a5: Neither blocks nor log segments found. Creating new log.
I20260620 09:26:09.731783 18180 tablet_bootstrap.cc:492] T dcb0cb99ed8c44f39b2a1f33929861b0 P 33d1881b38b9401b85c7fb2a3d976d24: No bootstrap required, opened a new log
I20260620 09:26:09.731856 18180 ts_tablet_manager.cc:1397] T dcb0cb99ed8c44f39b2a1f33929861b0 P 33d1881b38b9401b85c7fb2a3d976d24: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 09:26:09.732012 18180 raft_consensus.cc:348] T dcb0cb99ed8c44f39b2a1f33929861b0 P 33d1881b38b9401b85c7fb2a3d976d24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afc6a0bc3a714e809d81ab77676481a5" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 46835 } } peers { permanent_uuid: "7c582790769f481cbba1421f2cb1dc5e" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 43425 } } peers { permanent_uuid: "33d1881b38b9401b85c7fb2a3d976d24" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 33169 } }
I20260620 09:26:09.732093 18180 raft_consensus.cc:374] T dcb0cb99ed8c44f39b2a1f33929861b0 P 33d1881b38b9401b85c7fb2a3d976d24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:26:09.732087 18204 ts_tablet_manager.cc:1905] T e4e6cc7d88e54932bdd5fd56d1373794 P 7c582790769f481cbba1421f2cb1dc5e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:26:09.732118 18180 raft_consensus.cc:729] T dcb0cb99ed8c44f39b2a1f33929861b0 P 33d1881b38b9401b85c7fb2a3d976d24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33d1881b38b9401b85c7fb2a3d976d24, State: Initialized, Role: FOLLOWER
I20260620 09:26:09.732162 18180 consensus_queue.cc:260] T dcb0cb99ed8c44f39b2a1f33929861b0 P 33d1881b38b9401b85c7fb2a3d976d24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afc6a0bc3a714e809d81ab77676481a5" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 46835 } } peers { permanent_uuid: "7c582790769f481cbba1421f2cb1dc5e" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 43425 } } peers { permanent_uuid: "33d1881b38b9401b85c7fb2a3d976d24" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 33169 } }
I20260620 09:26:09.732196 18181 tablet_bootstrap.cc:492] T dcb0cb99ed8c44f39b2a1f33929861b0 P afc6a0bc3a714e809d81ab77676481a5: No bootstrap required, opened a new log
I20260620 09:26:09.732234 18181 ts_tablet_manager.cc:1397] T dcb0cb99ed8c44f39b2a1f33929861b0 P afc6a0bc3a714e809d81ab77676481a5: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260620 09:26:09.732263 18180 ts_tablet_manager.cc:1428] T dcb0cb99ed8c44f39b2a1f33929861b0 P 33d1881b38b9401b85c7fb2a3d976d24: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:09.732372 18181 raft_consensus.cc:348] T dcb0cb99ed8c44f39b2a1f33929861b0 P afc6a0bc3a714e809d81ab77676481a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afc6a0bc3a714e809d81ab77676481a5" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 46835 } } peers { permanent_uuid: "7c582790769f481cbba1421f2cb1dc5e" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 43425 } } peers { permanent_uuid: "33d1881b38b9401b85c7fb2a3d976d24" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 33169 } }
I20260620 09:26:09.732460 18181 raft_consensus.cc:374] T dcb0cb99ed8c44f39b2a1f33929861b0 P afc6a0bc3a714e809d81ab77676481a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:26:09.732486 18181 raft_consensus.cc:729] T dcb0cb99ed8c44f39b2a1f33929861b0 P afc6a0bc3a714e809d81ab77676481a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: afc6a0bc3a714e809d81ab77676481a5, State: Initialized, Role: FOLLOWER
I20260620 09:26:09.732529 18181 consensus_queue.cc:260] T dcb0cb99ed8c44f39b2a1f33929861b0 P afc6a0bc3a714e809d81ab77676481a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afc6a0bc3a714e809d81ab77676481a5" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 46835 } } peers { permanent_uuid: "7c582790769f481cbba1421f2cb1dc5e" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 43425 } } peers { permanent_uuid: "33d1881b38b9401b85c7fb2a3d976d24" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 33169 } }
I20260620 09:26:09.732630 18181 ts_tablet_manager.cc:1428] T dcb0cb99ed8c44f39b2a1f33929861b0 P afc6a0bc3a714e809d81ab77676481a5: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:09.733902 18204 ts_tablet_manager.cc:1918] T e4e6cc7d88e54932bdd5fd56d1373794 P 7c582790769f481cbba1421f2cb1dc5e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:26:09.733985 18204 log.cc:1196] T e4e6cc7d88e54932bdd5fd56d1373794 P 7c582790769f481cbba1421f2cb1dc5e: Deleting WAL directory at /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781947562949448-16558-0/minicluster-data/ts-1/wal/wals/e4e6cc7d88e54932bdd5fd56d1373794
I20260620 09:26:09.734239 18204 ts_tablet_manager.cc:1939] T e4e6cc7d88e54932bdd5fd56d1373794 P 7c582790769f481cbba1421f2cb1dc5e: Deleting consensus metadata
I20260620 09:26:09.734665 17660 catalog_manager.cc:4841] TS 7c582790769f481cbba1421f2cb1dc5e (127.16.43.130:43425): tablet e4e6cc7d88e54932bdd5fd56d1373794 (table test-table [id=bd4927c8842d4ae8ab4698c9579ca367]) successfully deleted
I20260620 09:26:09.798255 18191 raft_consensus.cc:482] T dcb0cb99ed8c44f39b2a1f33929861b0 P afc6a0bc3a714e809d81ab77676481a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:26:09.798372 18191 raft_consensus.cc:504] T dcb0cb99ed8c44f39b2a1f33929861b0 P afc6a0bc3a714e809d81ab77676481a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afc6a0bc3a714e809d81ab77676481a5" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 46835 } } peers { permanent_uuid: "7c582790769f481cbba1421f2cb1dc5e" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 43425 } } peers { permanent_uuid: "33d1881b38b9401b85c7fb2a3d976d24" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 33169 } }
I20260620 09:26:09.798592 18191 leader_election.cc:288] T dcb0cb99ed8c44f39b2a1f33929861b0 P afc6a0bc3a714e809d81ab77676481a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7c582790769f481cbba1421f2cb1dc5e (127.16.43.130:43425), 33d1881b38b9401b85c7fb2a3d976d24 (127.16.43.131:33169)
I20260620 09:26:09.798802 18111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcb0cb99ed8c44f39b2a1f33929861b0" candidate_uuid: "afc6a0bc3a714e809d81ab77676481a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33d1881b38b9401b85c7fb2a3d976d24" is_pre_election: true
I20260620 09:26:09.798823 17971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcb0cb99ed8c44f39b2a1f33929861b0" candidate_uuid: "afc6a0bc3a714e809d81ab77676481a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c582790769f481cbba1421f2cb1dc5e" is_pre_election: true
I20260620 09:26:09.798900 18111 raft_consensus.cc:2454] T dcb0cb99ed8c44f39b2a1f33929861b0 P 33d1881b38b9401b85c7fb2a3d976d24 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate afc6a0bc3a714e809d81ab77676481a5 in term 0.
I20260620 09:26:09.798900 17971 raft_consensus.cc:2454] T dcb0cb99ed8c44f39b2a1f33929861b0 P 7c582790769f481cbba1421f2cb1dc5e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate afc6a0bc3a714e809d81ab77676481a5 in term 0.
I20260620 09:26:09.799084 17762 leader_election.cc:302] T dcb0cb99ed8c44f39b2a1f33929861b0 P afc6a0bc3a714e809d81ab77676481a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33d1881b38b9401b85c7fb2a3d976d24, afc6a0bc3a714e809d81ab77676481a5; no voters:
I20260620 09:26:09.799163 18191 raft_consensus.cc:2781] T dcb0cb99ed8c44f39b2a1f33929861b0 P afc6a0bc3a714e809d81ab77676481a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 09:26:09.799202 18191 raft_consensus.cc:482] T dcb0cb99ed8c44f39b2a1f33929861b0 P afc6a0bc3a714e809d81ab77676481a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 09:26:09.799222 18191 raft_consensus.cc:3037] T dcb0cb99ed8c44f39b2a1f33929861b0 P afc6a0bc3a714e809d81ab77676481a5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:26:09.799954 18191 raft_consensus.cc:504] T dcb0cb99ed8c44f39b2a1f33929861b0 P afc6a0bc3a714e809d81ab77676481a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afc6a0bc3a714e809d81ab77676481a5" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 46835 } } peers { permanent_uuid: "7c582790769f481cbba1421f2cb1dc5e" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 43425 } } peers { permanent_uuid: "33d1881b38b9401b85c7fb2a3d976d24" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 33169 } }
I20260620 09:26:09.800092 18191 leader_election.cc:288] T dcb0cb99ed8c44f39b2a1f33929861b0 P afc6a0bc3a714e809d81ab77676481a5 [CANDIDATE]: Term 1 election: Requested vote from peers 7c582790769f481cbba1421f2cb1dc5e (127.16.43.130:43425), 33d1881b38b9401b85c7fb2a3d976d24 (127.16.43.131:33169)
I20260620 09:26:09.800225 17971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcb0cb99ed8c44f39b2a1f33929861b0" candidate_uuid: "afc6a0bc3a714e809d81ab77676481a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c582790769f481cbba1421f2cb1dc5e"
I20260620 09:26:09.800302 17971 raft_consensus.cc:3037] T dcb0cb99ed8c44f39b2a1f33929861b0 P 7c582790769f481cbba1421f2cb1dc5e [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:26:09.800320 18111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcb0cb99ed8c44f39b2a1f33929861b0" candidate_uuid: "afc6a0bc3a714e809d81ab77676481a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33d1881b38b9401b85c7fb2a3d976d24"
I20260620 09:26:09.800433 18111 raft_consensus.cc:3037] T dcb0cb99ed8c44f39b2a1f33929861b0 P 33d1881b38b9401b85c7fb2a3d976d24 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:26:09.801031 17971 raft_consensus.cc:2454] T dcb0cb99ed8c44f39b2a1f33929861b0 P 7c582790769f481cbba1421f2cb1dc5e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate afc6a0bc3a714e809d81ab77676481a5 in term 1.
I20260620 09:26:09.801214 17764 leader_election.cc:302] T dcb0cb99ed8c44f39b2a1f33929861b0 P afc6a0bc3a714e809d81ab77676481a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c582790769f481cbba1421f2cb1dc5e, afc6a0bc3a714e809d81ab77676481a5; no voters:
I20260620 09:26:09.801249 18111 raft_consensus.cc:2454] T dcb0cb99ed8c44f39b2a1f33929861b0 P 33d1881b38b9401b85c7fb2a3d976d24 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate afc6a0bc3a714e809d81ab77676481a5 in term 1.
I20260620 09:26:09.801326 18191 raft_consensus.cc:2781] T dcb0cb99ed8c44f39b2a1f33929861b0 P afc6a0bc3a714e809d81ab77676481a5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:26:09.801369 18191 raft_consensus.cc:686] T dcb0cb99ed8c44f39b2a1f33929861b0 P afc6a0bc3a714e809d81ab77676481a5 [term 1 LEADER]: Becoming Leader. State: Replica: afc6a0bc3a714e809d81ab77676481a5, State: Running, Role: LEADER
I20260620 09:26:09.801422 18191 consensus_queue.cc:237] T dcb0cb99ed8c44f39b2a1f33929861b0 P afc6a0bc3a714e809d81ab77676481a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afc6a0bc3a714e809d81ab77676481a5" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 46835 } } peers { permanent_uuid: "7c582790769f481cbba1421f2cb1dc5e" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 43425 } } peers { permanent_uuid: "33d1881b38b9401b85c7fb2a3d976d24" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 33169 } }
I20260620 09:26:09.801970 17673 catalog_manager.cc:5495] T dcb0cb99ed8c44f39b2a1f33929861b0 P afc6a0bc3a714e809d81ab77676481a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to afc6a0bc3a714e809d81ab77676481a5 (127.16.43.129). New cstate: current_term: 1 leader_uuid: "afc6a0bc3a714e809d81ab77676481a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afc6a0bc3a714e809d81ab77676481a5" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 46835 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7c582790769f481cbba1421f2cb1dc5e" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 43425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "33d1881b38b9401b85c7fb2a3d976d24" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 33169 } health_report { overall_health: UNKNOWN } } }
Jun 20 09:26:09 dist-test-slave-706q krb5kdc[17629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947568, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.190@KRBTEST.COM for kudu/127.16.43.190@KRBTEST.COM
I20260620 09:26:09.828634 18217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:09.820369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51268 (local address 127.16.43.190:42073)
0620 09:26:09.820533 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:09.820535 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:09.820546 (+ 11us) server_negotiation.cc:407] Connection header received
0620 09:26:09.820605 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:09.820606 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:09.820616 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:09.820697 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:09.821280 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:09.821900 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:09.822895 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:09.823127 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:09.823188 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:09.825752 (+ 2564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:09.825754 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:09.825755 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:26:09.825760 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:09.827381 (+ 1621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:09.827852 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:09.827853 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:09.827853 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:09.827918 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:09.828138 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:09.828139 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:09.828139 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:09.828249 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:09.828317 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:09.828478 (+ 161us) server_negotiation.cc:299] Negotiation successful
0620 09:26:09.828558 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":98,"threads_started":1}
I20260620 09:26:09.828748 18216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:09.820273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:42073 (local address 127.0.0.1:51268)
0620 09:26:09.820458 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:09.820471 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:09.820538 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:09.820727 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:09.820728 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:09.821129 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:09.821187 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:09.821189 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:09.821992 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:09.821994 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:09.822791 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:09.822793 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:09.823212 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:09.823213 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:09.823276 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:09.823996 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:26:09.824001 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:26:09.825670 (+ 1669us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:26:09.827474 (+ 1804us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:09.827482 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:09.827485 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:09.827769 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:09.827996 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:09.827997 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:09.827998 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:09.828071 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:09.828316 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:26:09.828323 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:26:09.828375 (+ 52us) client_negotiation.cc:769] Sending connection context
0620 09:26:09.828452 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 09:26:09.828646 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"mutex_wait_us":79,"thread_start_us":79,"threads_started":1}
I20260620 09:26:09.831162 17673 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.16.43.190@KRBTEST.COM'} at 127.0.0.1:51268:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 09:26:09.831391 17673 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 09:26:09.832894 18091 tablet_service.cc:1467] Processing CreateTablet for tablet 090895666e6a423797a58243948473a0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=fad0461d3f694fb999fc3fba960adbe1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 09:26:09.832894 17951 tablet_service.cc:1467] Processing CreateTablet for tablet 090895666e6a423797a58243948473a0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=fad0461d3f694fb999fc3fba960adbe1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 09:26:09.832921 17811 tablet_service.cc:1467] Processing CreateTablet for tablet 090895666e6a423797a58243948473a0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=fad0461d3f694fb999fc3fba960adbe1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 09:26:09.833050 18091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 090895666e6a423797a58243948473a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:26:09.833048 17951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 090895666e6a423797a58243948473a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:26:09.833083 17811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 090895666e6a423797a58243948473a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:26:09.834574 18182 tablet_bootstrap.cc:492] T 090895666e6a423797a58243948473a0 P 7c582790769f481cbba1421f2cb1dc5e: Bootstrap starting.
I20260620 09:26:09.834574 18181 tablet_bootstrap.cc:492] T 090895666e6a423797a58243948473a0 P afc6a0bc3a714e809d81ab77676481a5: Bootstrap starting.
I20260620 09:26:09.834944 18181 tablet_bootstrap.cc:654] T 090895666e6a423797a58243948473a0 P afc6a0bc3a714e809d81ab77676481a5: Neither blocks nor log segments found. Creating new log.
I20260620 09:26:09.834995 18182 tablet_bootstrap.cc:654] T 090895666e6a423797a58243948473a0 P 7c582790769f481cbba1421f2cb1dc5e: Neither blocks nor log segments found. Creating new log.
I20260620 09:26:09.835332 18180 tablet_bootstrap.cc:492] T 090895666e6a423797a58243948473a0 P 33d1881b38b9401b85c7fb2a3d976d24: Bootstrap starting.
I20260620 09:26:09.836001 18180 tablet_bootstrap.cc:654] T 090895666e6a423797a58243948473a0 P 33d1881b38b9401b85c7fb2a3d976d24: Neither blocks nor log segments found. Creating new log.
I20260620 09:26:09.836477 18181 tablet_bootstrap.cc:492] T 090895666e6a423797a58243948473a0 P afc6a0bc3a714e809d81ab77676481a5: No bootstrap required, opened a new log
I20260620 09:26:09.836529 18181 ts_tablet_manager.cc:1397] T 090895666e6a423797a58243948473a0 P afc6a0bc3a714e809d81ab77676481a5: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 09:26:09.836689 18181 raft_consensus.cc:348] T 090895666e6a423797a58243948473a0 P afc6a0bc3a714e809d81ab77676481a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afc6a0bc3a714e809d81ab77676481a5" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 46835 } } peers { permanent_uuid: "7c582790769f481cbba1421f2cb1dc5e" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 43425 } } peers { permanent_uuid: "33d1881b38b9401b85c7fb2a3d976d24" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 33169 } }
I20260620 09:26:09.836766 18181 raft_consensus.cc:374] T 090895666e6a423797a58243948473a0 P afc6a0bc3a714e809d81ab77676481a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:26:09.836791 18181 raft_consensus.cc:729] T 090895666e6a423797a58243948473a0 P afc6a0bc3a714e809d81ab77676481a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: afc6a0bc3a714e809d81ab77676481a5, State: Initialized, Role: FOLLOWER
I20260620 09:26:09.836841 18181 consensus_queue.cc:260] T 090895666e6a423797a58243948473a0 P afc6a0bc3a714e809d81ab77676481a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afc6a0bc3a714e809d81ab77676481a5" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 46835 } } peers { permanent_uuid: "7c582790769f481cbba1421f2cb1dc5e" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 43425 } } peers { permanent_uuid: "33d1881b38b9401b85c7fb2a3d976d24" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 33169 } }
I20260620 09:26:09.836935 18181 ts_tablet_manager.cc:1428] T 090895666e6a423797a58243948473a0 P afc6a0bc3a714e809d81ab77676481a5: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:09.837023 18191 tablet_replica.cc:439] T 090895666e6a423797a58243948473a0 P afc6a0bc3a714e809d81ab77676481a5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afc6a0bc3a714e809d81ab77676481a5" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 46835 } } peers { permanent_uuid: "7c582790769f481cbba1421f2cb1dc5e" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 43425 } } peers { permanent_uuid: "33d1881b38b9401b85c7fb2a3d976d24" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 33169 } } }
I20260620 09:26:09.837105 18191 tablet_replica.cc:442] T 090895666e6a423797a58243948473a0 P afc6a0bc3a714e809d81ab77676481a5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 09:26:09.837316 18180 tablet_bootstrap.cc:492] T 090895666e6a423797a58243948473a0 P 33d1881b38b9401b85c7fb2a3d976d24: No bootstrap required, opened a new log
I20260620 09:26:09.837363 18180 ts_tablet_manager.cc:1397] T 090895666e6a423797a58243948473a0 P 33d1881b38b9401b85c7fb2a3d976d24: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 09:26:09.837514 18180 raft_consensus.cc:348] T 090895666e6a423797a58243948473a0 P 33d1881b38b9401b85c7fb2a3d976d24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afc6a0bc3a714e809d81ab77676481a5" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 46835 } } peers { permanent_uuid: "7c582790769f481cbba1421f2cb1dc5e" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 43425 } } peers { permanent_uuid: "33d1881b38b9401b85c7fb2a3d976d24" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 33169 } }
I20260620 09:26:09.837587 18180 raft_consensus.cc:374] T 090895666e6a423797a58243948473a0 P 33d1881b38b9401b85c7fb2a3d976d24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:26:09.837615 18180 raft_consensus.cc:729] T 090895666e6a423797a58243948473a0 P 33d1881b38b9401b85c7fb2a3d976d24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33d1881b38b9401b85c7fb2a3d976d24, State: Initialized, Role: FOLLOWER
I20260620 09:26:09.837657 18180 consensus_queue.cc:260] T 090895666e6a423797a58243948473a0 P 33d1881b38b9401b85c7fb2a3d976d24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afc6a0bc3a714e809d81ab77676481a5" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 46835 } } peers { permanent_uuid: "7c582790769f481cbba1421f2cb1dc5e" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 43425 } } peers { permanent_uuid: "33d1881b38b9401b85c7fb2a3d976d24" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 33169 } }
I20260620 09:26:09.837750 18180 ts_tablet_manager.cc:1428] T 090895666e6a423797a58243948473a0 P 33d1881b38b9401b85c7fb2a3d976d24: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260620 09:26:09.837893 18187 tablet_replica.cc:439] T 090895666e6a423797a58243948473a0 P 33d1881b38b9401b85c7fb2a3d976d24: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afc6a0bc3a714e809d81ab77676481a5" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 46835 } } peers { permanent_uuid: "7c582790769f481cbba1421f2cb1dc5e" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 43425 } } peers { permanent_uuid: "33d1881b38b9401b85c7fb2a3d976d24" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 33169 } } }
I20260620 09:26:09.837973 18187 tablet_replica.cc:442] T 090895666e6a423797a58243948473a0 P 33d1881b38b9401b85c7fb2a3d976d24: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 09:26:09.838416 18182 tablet_bootstrap.cc:492] T 090895666e6a423797a58243948473a0 P 7c582790769f481cbba1421f2cb1dc5e: No bootstrap required, opened a new log
I20260620 09:26:09.838475 18182 ts_tablet_manager.cc:1397] T 090895666e6a423797a58243948473a0 P 7c582790769f481cbba1421f2cb1dc5e: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260620 09:26:09.838622 18182 raft_consensus.cc:348] T 090895666e6a423797a58243948473a0 P 7c582790769f481cbba1421f2cb1dc5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afc6a0bc3a714e809d81ab77676481a5" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 46835 } } peers { permanent_uuid: "7c582790769f481cbba1421f2cb1dc5e" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 43425 } } peers { permanent_uuid: "33d1881b38b9401b85c7fb2a3d976d24" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 33169 } }
I20260620 09:26:09.838701 18182 raft_consensus.cc:374] T 090895666e6a423797a58243948473a0 P 7c582790769f481cbba1421f2cb1dc5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:26:09.838732 18182 raft_consensus.cc:729] T 090895666e6a423797a58243948473a0 P 7c582790769f481cbba1421f2cb1dc5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c582790769f481cbba1421f2cb1dc5e, State: Initialized, Role: FOLLOWER
I20260620 09:26:09.838781 18182 consensus_queue.cc:260] T 090895666e6a423797a58243948473a0 P 7c582790769f481cbba1421f2cb1dc5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afc6a0bc3a714e809d81ab77676481a5" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 46835 } } peers { permanent_uuid: "7c582790769f481cbba1421f2cb1dc5e" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 43425 } } peers { permanent_uuid: "33d1881b38b9401b85c7fb2a3d976d24" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 33169 } }
I20260620 09:26:09.838874 18182 ts_tablet_manager.cc:1428] T 090895666e6a423797a58243948473a0 P 7c582790769f481cbba1421f2cb1dc5e: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 09:26:09.838954 18188 tablet_replica.cc:439] T 090895666e6a423797a58243948473a0 P 7c582790769f481cbba1421f2cb1dc5e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afc6a0bc3a714e809d81ab77676481a5" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 46835 } } peers { permanent_uuid: "7c582790769f481cbba1421f2cb1dc5e" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 43425 } } peers { permanent_uuid: "33d1881b38b9401b85c7fb2a3d976d24" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 33169 } } }
I20260620 09:26:09.839051 18188 tablet_replica.cc:442] T 090895666e6a423797a58243948473a0 P 7c582790769f481cbba1421f2cb1dc5e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 09:26:09.877755 18187 raft_consensus.cc:482] T 090895666e6a423797a58243948473a0 P 33d1881b38b9401b85c7fb2a3d976d24 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:26:09.877858 18187 raft_consensus.cc:504] T 090895666e6a423797a58243948473a0 P 33d1881b38b9401b85c7fb2a3d976d24 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afc6a0bc3a714e809d81ab77676481a5" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 46835 } } peers { permanent_uuid: "7c582790769f481cbba1421f2cb1dc5e" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 43425 } } peers { permanent_uuid: "33d1881b38b9401b85c7fb2a3d976d24" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 33169 } }
I20260620 09:26:09.878201 18187 leader_election.cc:288] T 090895666e6a423797a58243948473a0 P 33d1881b38b9401b85c7fb2a3d976d24 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers afc6a0bc3a714e809d81ab77676481a5 (127.16.43.129:46835), 7c582790769f481cbba1421f2cb1dc5e (127.16.43.130:43425)
I20260620 09:26:09.881466 18176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:09.878508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.131:47393 (local address 127.16.43.130:43425)
0620 09:26:09.878574 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:09.878576 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:09.878606 (+ 30us) server_negotiation.cc:407] Connection header received
0620 09:26:09.878709 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:09.878710 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:09.878720 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:09.878798 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:26:09.878972 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:09.879439 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:09.880734 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:09.881082 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:09.881171 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:09.881181 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:09.881338 (+ 157us) server_negotiation.cc:299] Negotiation successful
0620 09:26:09.881374 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 09:26:09.881480 18177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:09.878476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.131:39827 (local address 127.16.43.129:46835)
0620 09:26:09.878528 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:09.878531 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:09.878620 (+ 89us) server_negotiation.cc:407] Connection header received
0620 09:26:09.878699 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:09.878700 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:09.878716 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:09.878776 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:26:09.878932 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:09.879408 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:09.880746 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:09.881109 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:09.881192 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:09.881200 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:09.881361 (+ 161us) server_negotiation.cc:299] Negotiation successful
0620 09:26:09.881400 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 09:26:09.881497 18222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:09.878360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.129:46835 (local address 127.16.43.131:39827)
0620 09:26:09.878534 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:09.878546 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:09.878631 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:09.878781 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:09.878781 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:09.878783 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:26:09.878851 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:09.878853 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:09.879540 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:09.879541 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:09.880651 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:09.880653 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:09.881191 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:09.881191 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:09.881273 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:09.881276 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 09:26:09.881352 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 09:26:09.881393 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"mutex_wait_us":54,"thread_start_us":95,"threads_started":1}
I20260620 09:26:09.881502 18221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:09.878413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.130:43425 (local address 127.16.43.131:47393)
0620 09:26:09.878540 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:09.878551 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:09.878626 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:09.878802 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:09.878803 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:09.878805 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:26:09.878887 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:09.878888 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:09.879553 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:09.879554 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:09.880645 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:09.880647 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:09.881180 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:09.881181 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:09.881255 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:09.881259 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 09:26:09.881332 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 09:26:09.881373 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"mutex_wait_us":31,"thread_start_us":59,"threads_started":1}
I20260620 09:26:09.881785 17821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "090895666e6a423797a58243948473a0" candidate_uuid: "33d1881b38b9401b85c7fb2a3d976d24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "afc6a0bc3a714e809d81ab77676481a5" is_pre_election: true
I20260620 09:26:09.881918 17821 raft_consensus.cc:2454] T 090895666e6a423797a58243948473a0 P afc6a0bc3a714e809d81ab77676481a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 33d1881b38b9401b85c7fb2a3d976d24 in term 0.
I20260620 09:26:09.881995 17971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "090895666e6a423797a58243948473a0" candidate_uuid: "33d1881b38b9401b85c7fb2a3d976d24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c582790769f481cbba1421f2cb1dc5e" is_pre_election: true
I20260620 09:26:09.882081 17971 raft_consensus.cc:2454] T 090895666e6a423797a58243948473a0 P 7c582790769f481cbba1421f2cb1dc5e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 33d1881b38b9401b85c7fb2a3d976d24 in term 0.
I20260620 09:26:09.882105 18042 leader_election.cc:302] T 090895666e6a423797a58243948473a0 P 33d1881b38b9401b85c7fb2a3d976d24 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33d1881b38b9401b85c7fb2a3d976d24, afc6a0bc3a714e809d81ab77676481a5; no voters:
I20260620 09:26:09.882226 18187 raft_consensus.cc:2781] T 090895666e6a423797a58243948473a0 P 33d1881b38b9401b85c7fb2a3d976d24 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 09:26:09.882261 18187 raft_consensus.cc:482] T 090895666e6a423797a58243948473a0 P 33d1881b38b9401b85c7fb2a3d976d24 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 09:26:09.882285 18187 raft_consensus.cc:3037] T 090895666e6a423797a58243948473a0 P 33d1881b38b9401b85c7fb2a3d976d24 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:26:09.882987 18187 raft_consensus.cc:504] T 090895666e6a423797a58243948473a0 P 33d1881b38b9401b85c7fb2a3d976d24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afc6a0bc3a714e809d81ab77676481a5" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 46835 } } peers { permanent_uuid: "7c582790769f481cbba1421f2cb1dc5e" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 43425 } } peers { permanent_uuid: "33d1881b38b9401b85c7fb2a3d976d24" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 33169 } }
I20260620 09:26:09.883136 18187 leader_election.cc:288] T 090895666e6a423797a58243948473a0 P 33d1881b38b9401b85c7fb2a3d976d24 [CANDIDATE]: Term 1 election: Requested vote from peers afc6a0bc3a714e809d81ab77676481a5 (127.16.43.129:46835), 7c582790769f481cbba1421f2cb1dc5e (127.16.43.130:43425)
I20260620 09:26:09.883298 17971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "090895666e6a423797a58243948473a0" candidate_uuid: "33d1881b38b9401b85c7fb2a3d976d24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c582790769f481cbba1421f2cb1dc5e"
I20260620 09:26:09.883298 17821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "090895666e6a423797a58243948473a0" candidate_uuid: "33d1881b38b9401b85c7fb2a3d976d24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "afc6a0bc3a714e809d81ab77676481a5"
I20260620 09:26:09.883376 17821 raft_consensus.cc:3037] T 090895666e6a423797a58243948473a0 P afc6a0bc3a714e809d81ab77676481a5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:26:09.883375 17971 raft_consensus.cc:3037] T 090895666e6a423797a58243948473a0 P 7c582790769f481cbba1421f2cb1dc5e [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:26:09.884133 17971 raft_consensus.cc:2454] T 090895666e6a423797a58243948473a0 P 7c582790769f481cbba1421f2cb1dc5e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33d1881b38b9401b85c7fb2a3d976d24 in term 1.
I20260620 09:26:09.884208 17821 raft_consensus.cc:2454] T 090895666e6a423797a58243948473a0 P afc6a0bc3a714e809d81ab77676481a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33d1881b38b9401b85c7fb2a3d976d24 in term 1.
I20260620 09:26:09.884331 18044 leader_election.cc:302] T 090895666e6a423797a58243948473a0 P 33d1881b38b9401b85c7fb2a3d976d24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33d1881b38b9401b85c7fb2a3d976d24, 7c582790769f481cbba1421f2cb1dc5e; no voters:
I20260620 09:26:09.884469 18187 raft_consensus.cc:2781] T 090895666e6a423797a58243948473a0 P 33d1881b38b9401b85c7fb2a3d976d24 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:26:09.884624 18187 raft_consensus.cc:686] T 090895666e6a423797a58243948473a0 P 33d1881b38b9401b85c7fb2a3d976d24 [term 1 LEADER]: Becoming Leader. State: Replica: 33d1881b38b9401b85c7fb2a3d976d24, State: Running, Role: LEADER
I20260620 09:26:09.884706 18187 consensus_queue.cc:237] T 090895666e6a423797a58243948473a0 P 33d1881b38b9401b85c7fb2a3d976d24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afc6a0bc3a714e809d81ab77676481a5" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 46835 } } peers { permanent_uuid: "7c582790769f481cbba1421f2cb1dc5e" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 43425 } } peers { permanent_uuid: "33d1881b38b9401b85c7fb2a3d976d24" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 33169 } }
I20260620 09:26:09.885115 18223 tablet_replica.cc:439] T 090895666e6a423797a58243948473a0 P 33d1881b38b9401b85c7fb2a3d976d24: TxnStatusTablet state changed. Reason: New leader 33d1881b38b9401b85c7fb2a3d976d24. Latest consensus state: current_term: 1 leader_uuid: "33d1881b38b9401b85c7fb2a3d976d24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afc6a0bc3a714e809d81ab77676481a5" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 46835 } } peers { permanent_uuid: "7c582790769f481cbba1421f2cb1dc5e" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 43425 } } peers { permanent_uuid: "33d1881b38b9401b85c7fb2a3d976d24" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 33169 } } }
I20260620 09:26:09.885206 18223 tablet_replica.cc:442] T 090895666e6a423797a58243948473a0 P 33d1881b38b9401b85c7fb2a3d976d24: This TxnStatusTablet replica's current role is: LEADER
I20260620 09:26:09.885391 18225 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 09:26:09.885519 18225 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 09:26:09.885561 17673 catalog_manager.cc:5495] T 090895666e6a423797a58243948473a0 P 33d1881b38b9401b85c7fb2a3d976d24 reported cstate change: term changed from 0 to 1, leader changed from <none> to 33d1881b38b9401b85c7fb2a3d976d24 (127.16.43.131). New cstate: current_term: 1 leader_uuid: "33d1881b38b9401b85c7fb2a3d976d24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afc6a0bc3a714e809d81ab77676481a5" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 46835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c582790769f481cbba1421f2cb1dc5e" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 43425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "33d1881b38b9401b85c7fb2a3d976d24" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 33169 } health_report { overall_health: HEALTHY } } }
I20260620 09:26:09.897238 18216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:09.894379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.131:33169 (local address 127.0.0.1:41744)
0620 09:26:09.894469 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:09.894483 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:09.894536 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:09.894716 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:09.894717 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:09.894718 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 09:26:09.894770 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:09.894772 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:09.895700 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:09.895701 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:09.896279 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:09.896281 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:09.896744 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:09.896745 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:09.896790 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:09.896805 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 09:26:09.897092 (+ 287us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 09:26:09.897099 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 09:26:09.897137 (+ 38us) client_negotiation.cc:240] Negotiation successful
0620 09:26:09.897175 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260620 09:26:09.897300 18179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:09.894477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41744 (local address 127.16.43.131:33169)
0620 09:26:09.894545 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:09.894547 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:09.894557 (+ 10us) server_negotiation.cc:407] Connection header received
0620 09:26:09.894615 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:09.894616 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:09.894638 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:09.894716 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:26:09.894833 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:09.895611 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:09.896359 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:09.896667 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:09.896737 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:09.896891 (+ 154us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:26:09.897021 (+ 130us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:26:09.897102 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:09.897146 (+ 44us) server_negotiation.cc:299] Negotiation successful
0620 09:26:09.897174 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 09:26:09.898509 17971 raft_consensus.cc:1261] T 090895666e6a423797a58243948473a0 P 7c582790769f481cbba1421f2cb1dc5e [term 1 FOLLOWER]: Refusing update from remote peer 33d1881b38b9401b85c7fb2a3d976d24: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:26:09.898609 17821 raft_consensus.cc:1261] T 090895666e6a423797a58243948473a0 P afc6a0bc3a714e809d81ab77676481a5 [term 1 FOLLOWER]: Refusing update from remote peer 33d1881b38b9401b85c7fb2a3d976d24: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:26:09.898782 18223 consensus_queue.cc:1035] T 090895666e6a423797a58243948473a0 P 33d1881b38b9401b85c7fb2a3d976d24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c582790769f481cbba1421f2cb1dc5e" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 43425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:26:09.898975 18187 consensus_queue.cc:1035] T 090895666e6a423797a58243948473a0 P 33d1881b38b9401b85c7fb2a3d976d24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "afc6a0bc3a714e809d81ab77676481a5" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 46835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:26:09.899617 18188 tablet_replica.cc:439] T 090895666e6a423797a58243948473a0 P 7c582790769f481cbba1421f2cb1dc5e: TxnStatusTablet state changed. Reason: New leader 33d1881b38b9401b85c7fb2a3d976d24. Latest consensus state: current_term: 1 leader_uuid: "33d1881b38b9401b85c7fb2a3d976d24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afc6a0bc3a714e809d81ab77676481a5" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 46835 } } peers { permanent_uuid: "7c582790769f481cbba1421f2cb1dc5e" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 43425 } } peers { permanent_uuid: "33d1881b38b9401b85c7fb2a3d976d24" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 33169 } } }
I20260620 09:26:09.899629 18191 tablet_replica.cc:439] T 090895666e6a423797a58243948473a0 P afc6a0bc3a714e809d81ab77676481a5: TxnStatusTablet state changed. Reason: New leader 33d1881b38b9401b85c7fb2a3d976d24. Latest consensus state: current_term: 1 leader_uuid: "33d1881b38b9401b85c7fb2a3d976d24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afc6a0bc3a714e809d81ab77676481a5" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 46835 } } peers { permanent_uuid: "7c582790769f481cbba1421f2cb1dc5e" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 43425 } } peers { permanent_uuid: "33d1881b38b9401b85c7fb2a3d976d24" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 33169 } } }
I20260620 09:26:09.899720 18191 tablet_replica.cc:442] T 090895666e6a423797a58243948473a0 P afc6a0bc3a714e809d81ab77676481a5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 09:26:09.899722 18188 tablet_replica.cc:442] T 090895666e6a423797a58243948473a0 P 7c582790769f481cbba1421f2cb1dc5e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 09:26:09.900517 18196 mvcc.cc:205] Tried to move back new op lower bound from 7298857246302199808 to 7298857246248542208. Current Snapshot: MvccSnapshot[applied={T|T < 7298857246302199808}]
I20260620 09:26:09.900591 18187 tablet_replica.cc:439] T 090895666e6a423797a58243948473a0 P 33d1881b38b9401b85c7fb2a3d976d24: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "33d1881b38b9401b85c7fb2a3d976d24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afc6a0bc3a714e809d81ab77676481a5" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 46835 } } peers { permanent_uuid: "7c582790769f481cbba1421f2cb1dc5e" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 43425 } } peers { permanent_uuid: "33d1881b38b9401b85c7fb2a3d976d24" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 33169 } } }
I20260620 09:26:09.900700 18187 tablet_replica.cc:442] T 090895666e6a423797a58243948473a0 P 33d1881b38b9401b85c7fb2a3d976d24: This TxnStatusTablet replica's current role is: LEADER
I20260620 09:26:09.900974 18187 tablet_replica.cc:439] T 090895666e6a423797a58243948473a0 P 33d1881b38b9401b85c7fb2a3d976d24: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "33d1881b38b9401b85c7fb2a3d976d24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afc6a0bc3a714e809d81ab77676481a5" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 46835 } } peers { permanent_uuid: "7c582790769f481cbba1421f2cb1dc5e" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 43425 } } peers { permanent_uuid: "33d1881b38b9401b85c7fb2a3d976d24" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 33169 } } }
I20260620 09:26:09.901047 18187 tablet_replica.cc:442] T 090895666e6a423797a58243948473a0 P 33d1881b38b9401b85c7fb2a3d976d24: This TxnStatusTablet replica's current role is: LEADER
I20260620 09:26:09.901178 18188 tablet_replica.cc:439] T 090895666e6a423797a58243948473a0 P 7c582790769f481cbba1421f2cb1dc5e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "33d1881b38b9401b85c7fb2a3d976d24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afc6a0bc3a714e809d81ab77676481a5" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 46835 } } peers { permanent_uuid: "7c582790769f481cbba1421f2cb1dc5e" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 43425 } } peers { permanent_uuid: "33d1881b38b9401b85c7fb2a3d976d24" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 33169 } } }
I20260620 09:26:09.901268 18188 tablet_replica.cc:442] T 090895666e6a423797a58243948473a0 P 7c582790769f481cbba1421f2cb1dc5e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 09:26:09.901409 18194 mvcc.cc:205] Tried to move back new op lower bound from 7298857246302199808 to 7298857246248542208. Current Snapshot: MvccSnapshot[applied={T|T < 7298857246302199808}]
I20260620 09:26:09.901492 18191 tablet_replica.cc:439] T 090895666e6a423797a58243948473a0 P afc6a0bc3a714e809d81ab77676481a5: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "33d1881b38b9401b85c7fb2a3d976d24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afc6a0bc3a714e809d81ab77676481a5" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 46835 } } peers { permanent_uuid: "7c582790769f481cbba1421f2cb1dc5e" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 43425 } } peers { permanent_uuid: "33d1881b38b9401b85c7fb2a3d976d24" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 33169 } } }
I20260620 09:26:09.901554 18191 tablet_replica.cc:442] T 090895666e6a423797a58243948473a0 P afc6a0bc3a714e809d81ab77676481a5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 09:26:09.907292 18179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:09.904807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.129:55347 (local address 127.16.43.131:33169)
0620 09:26:09.904851 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:09.904853 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:09.904989 (+ 136us) server_negotiation.cc:407] Connection header received
0620 09:26:09.905061 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:09.905062 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:09.905072 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:09.905152 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:26:09.905261 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:09.905747 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:09.906531 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:09.906792 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:09.906876 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:09.907061 (+ 185us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:26:09.907094 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:26:09.907147 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:09.907202 (+ 55us) server_negotiation.cc:299] Negotiation successful
0620 09:26:09.907222 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 09:26:09.907328 18232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:09.904749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.131:33169 (local address 127.16.43.129:55347)
0620 09:26:09.904936 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:09.904946 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:09.905004 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:09.905155 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:09.905156 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:09.905158 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 09:26:09.905208 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:09.905210 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:09.905834 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:09.905835 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:09.906428 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:09.906430 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:09.906878 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:09.906880 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:09.906940 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:09.906956 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 09:26:09.907154 (+ 198us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 09:26:09.907156 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 09:26:09.907201 (+ 45us) client_negotiation.cc:240] Negotiation successful
0620 09:26:09.907248 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":84,"threads_started":1}
I20260620 09:26:09.909878 17971 raft_consensus.cc:1261] T dcb0cb99ed8c44f39b2a1f33929861b0 P 7c582790769f481cbba1421f2cb1dc5e [term 1 FOLLOWER]: Refusing update from remote peer afc6a0bc3a714e809d81ab77676481a5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:26:09.910100 18199 consensus_queue.cc:1035] T dcb0cb99ed8c44f39b2a1f33929861b0 P afc6a0bc3a714e809d81ab77676481a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c582790769f481cbba1421f2cb1dc5e" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 43425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:26:09.910810 18111 raft_consensus.cc:1261] T dcb0cb99ed8c44f39b2a1f33929861b0 P 33d1881b38b9401b85c7fb2a3d976d24 [term 1 FOLLOWER]: Refusing update from remote peer afc6a0bc3a714e809d81ab77676481a5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:26:09.911129 18199 consensus_queue.cc:1035] T dcb0cb99ed8c44f39b2a1f33929861b0 P afc6a0bc3a714e809d81ab77676481a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33d1881b38b9401b85c7fb2a3d976d24" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 33169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:26:09.919642 18177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:09.916171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.131:60075 (local address 127.16.43.129:46835)
0620 09:26:09.916215 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:09.916217 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:09.917086 (+ 869us) server_negotiation.cc:407] Connection header received
0620 09:26:09.917150 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:09.917151 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:09.917160 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:09.917202 (+ 42us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:26:09.917331 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:09.917803 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:09.918971 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:09.919157 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:09.919212 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:09.919403 (+ 191us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:26:09.919430 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:26:09.919463 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:09.919512 (+ 49us) server_negotiation.cc:299] Negotiation successful
0620 09:26:09.919570 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 09:26:09.919682 18236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:09.916120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.129:46835 (local address 127.16.43.131:60075)
0620 09:26:09.916976 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:09.916988 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:09.917094 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:09.917206 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:09.917207 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:09.917218 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 09:26:09.917283 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:09.917285 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:09.917881 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:09.917882 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:09.918887 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:09.918890 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:09.919235 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:09.919236 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:09.919298 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:09.919314 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 09:26:09.919465 (+ 151us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 09:26:09.919467 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 09:26:09.919504 (+ 37us) client_negotiation.cc:240] Negotiation successful
0620 09:26:09.919571 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":816,"thread_start_us":102,"threads_started":1}
I20260620 09:26:09.925638 17674 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51264:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:26:09.925776 17674 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51264:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:26:09.926457 17674 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2325df10f30d4fafbf003522017927f9: Sending DeleteTablet for 3 replicas of tablet dcb0cb99ed8c44f39b2a1f33929861b0
I20260620 09:26:09.926736 17811 tablet_service.cc:1514] Processing DeleteTablet for tablet dcb0cb99ed8c44f39b2a1f33929861b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:26:09 UTC) from {username='kudu', principal='kudu/127.16.43.190@KRBTEST.COM'} at 127.0.0.1:56372
I20260620 09:26:09.926741 18091 tablet_service.cc:1514] Processing DeleteTablet for tablet dcb0cb99ed8c44f39b2a1f33929861b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:26:09 UTC) from {username='kudu', principal='kudu/127.16.43.190@KRBTEST.COM'} at 127.0.0.1:41734
I20260620 09:26:09.926847 18205 tablet_replica.cc:330] T dcb0cb99ed8c44f39b2a1f33929861b0 P afc6a0bc3a714e809d81ab77676481a5: stopping tablet replica
I20260620 09:26:09.926865 18206 tablet_replica.cc:330] T dcb0cb99ed8c44f39b2a1f33929861b0 P 33d1881b38b9401b85c7fb2a3d976d24: stopping tablet replica
I20260620 09:26:09.926909 18205 raft_consensus.cc:2229] T dcb0cb99ed8c44f39b2a1f33929861b0 P afc6a0bc3a714e809d81ab77676481a5 [term 1 LEADER]: Raft consensus shutting down.
I20260620 09:26:09.926921 18206 raft_consensus.cc:2229] T dcb0cb99ed8c44f39b2a1f33929861b0 P 33d1881b38b9401b85c7fb2a3d976d24 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:26:09.926954 18205 raft_consensus.cc:2258] T dcb0cb99ed8c44f39b2a1f33929861b0 P afc6a0bc3a714e809d81ab77676481a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:26:09.926957 18206 raft_consensus.cc:2258] T dcb0cb99ed8c44f39b2a1f33929861b0 P 33d1881b38b9401b85c7fb2a3d976d24 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:26:09.927302 18205 ts_tablet_manager.cc:1905] T dcb0cb99ed8c44f39b2a1f33929861b0 P afc6a0bc3a714e809d81ab77676481a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:26:09.927488 17951 tablet_service.cc:1514] Processing DeleteTablet for tablet dcb0cb99ed8c44f39b2a1f33929861b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:26:09 UTC) from {username='kudu', principal='kudu/127.16.43.190@KRBTEST.COM'} at 127.0.0.1:41884
I20260620 09:26:09.927618 18204 tablet_replica.cc:330] T dcb0cb99ed8c44f39b2a1f33929861b0 P 7c582790769f481cbba1421f2cb1dc5e: stopping tablet replica
I20260620 09:26:09.927661 18204 raft_consensus.cc:2229] T dcb0cb99ed8c44f39b2a1f33929861b0 P 7c582790769f481cbba1421f2cb1dc5e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:26:09.927691 18204 raft_consensus.cc:2258] T dcb0cb99ed8c44f39b2a1f33929861b0 P 7c582790769f481cbba1421f2cb1dc5e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:26:09.928056 18206 ts_tablet_manager.cc:1905] T dcb0cb99ed8c44f39b2a1f33929861b0 P 33d1881b38b9401b85c7fb2a3d976d24: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:26:09.929211 18205 ts_tablet_manager.cc:1918] T dcb0cb99ed8c44f39b2a1f33929861b0 P afc6a0bc3a714e809d81ab77676481a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 09:26:09.929270 18205 log.cc:1196] T dcb0cb99ed8c44f39b2a1f33929861b0 P afc6a0bc3a714e809d81ab77676481a5: Deleting WAL directory at /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781947562949448-16558-0/minicluster-data/ts-0/wal/wals/dcb0cb99ed8c44f39b2a1f33929861b0
I20260620 09:26:09.929476 18205 ts_tablet_manager.cc:1939] T dcb0cb99ed8c44f39b2a1f33929861b0 P afc6a0bc3a714e809d81ab77676481a5: Deleting consensus metadata
I20260620 09:26:09.929679 18206 ts_tablet_manager.cc:1918] T dcb0cb99ed8c44f39b2a1f33929861b0 P 33d1881b38b9401b85c7fb2a3d976d24: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 09:26:09.929730 18206 log.cc:1196] T dcb0cb99ed8c44f39b2a1f33929861b0 P 33d1881b38b9401b85c7fb2a3d976d24: Deleting WAL directory at /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781947562949448-16558-0/minicluster-data/ts-2/wal/wals/dcb0cb99ed8c44f39b2a1f33929861b0
I20260620 09:26:09.929773 18204 ts_tablet_manager.cc:1905] T dcb0cb99ed8c44f39b2a1f33929861b0 P 7c582790769f481cbba1421f2cb1dc5e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:26:09.929942 18206 ts_tablet_manager.cc:1939] T dcb0cb99ed8c44f39b2a1f33929861b0 P 33d1881b38b9401b85c7fb2a3d976d24: Deleting consensus metadata
I20260620 09:26:09.929949 17658 catalog_manager.cc:4841] TS afc6a0bc3a714e809d81ab77676481a5 (127.16.43.129:46835): tablet dcb0cb99ed8c44f39b2a1f33929861b0 (table test-table [id=29d93747acdf416cbac91274de79fd99]) successfully deleted
I20260620 09:26:09.930398 17658 catalog_manager.cc:4841] TS 33d1881b38b9401b85c7fb2a3d976d24 (127.16.43.131:33169): tablet dcb0cb99ed8c44f39b2a1f33929861b0 (table test-table [id=29d93747acdf416cbac91274de79fd99]) successfully deleted
I20260620 09:26:09.931401 18204 ts_tablet_manager.cc:1918] T dcb0cb99ed8c44f39b2a1f33929861b0 P 7c582790769f481cbba1421f2cb1dc5e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 09:26:09.931461 18204 log.cc:1196] T dcb0cb99ed8c44f39b2a1f33929861b0 P 7c582790769f481cbba1421f2cb1dc5e: Deleting WAL directory at /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781947562949448-16558-0/minicluster-data/ts-1/wal/wals/dcb0cb99ed8c44f39b2a1f33929861b0
I20260620 09:26:09.931710 18204 ts_tablet_manager.cc:1939] T dcb0cb99ed8c44f39b2a1f33929861b0 P 7c582790769f481cbba1421f2cb1dc5e: Deleting consensus metadata
I20260620 09:26:09.932140 17660 catalog_manager.cc:4841] TS 7c582790769f481cbba1421f2cb1dc5e (127.16.43.130:43425): tablet dcb0cb99ed8c44f39b2a1f33929861b0 (table test-table [id=29d93747acdf416cbac91274de79fd99]) successfully deleted
Jun 20 09:26:09 dist-test-slave-706q krb5kdc[17629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947569, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.43.129@KRBTEST.COM
I20260620 09:26:09.937951 18177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:09.927650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56396 (local address 127.16.43.129:46835)
0620 09:26:09.927690 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:09.927693 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:09.928011 (+ 318us) server_negotiation.cc:407] Connection header received
0620 09:26:09.928106 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:09.928107 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:09.928121 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:09.928153 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:09.928982 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:09.929750 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:09.931222 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:09.931690 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:09.931745 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:09.934976 (+ 3231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:09.934986 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:09.934986 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:26:09.934994 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:09.936774 (+ 1780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:09.937196 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:09.937198 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:09.937199 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:09.937254 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:09.937464 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:09.937465 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:09.937465 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:09.937598 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:09.937700 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:09.937817 (+ 117us) server_negotiation.cc:299] Negotiation successful
0620 09:26:09.937883 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
W20260620 09:26:09.938262 17871 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56396
I20260620 09:26:09.945585 18217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:09.939313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51284 (local address 127.16.43.190:42073)
0620 09:26:09.939366 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:09.939368 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:09.939378 (+ 10us) server_negotiation.cc:407] Connection header received
0620 09:26:09.939464 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:09.939465 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:09.939479 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:09.939562 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:09.940064 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:09.940591 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:09.941297 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:09.941521 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:09.941593 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:09.942815 (+ 1222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:09.942817 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:09.942817 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:26:09.942826 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:09.944362 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:09.944752 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:09.944753 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:09.944754 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:09.944815 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:09.945035 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:09.945036 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:09.945037 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:09.945162 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:09.945277 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:09.945371 (+ 94us) server_negotiation.cc:299] Negotiation successful
0620 09:26:09.945463 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
W20260620 09:26:09.945875 17674 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51284
I20260620 09:26:09.946528 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 17746
I20260620 09:26:09.954635 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 17886
I20260620 09:26:09.960346 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 18026
I20260620 09:26:09.965461 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 17642
2026-06-20T09:26:09Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2578 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781947562949448-16558-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:26:10 dist-test-slave-706q krb5kdc[18244](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:26:10 dist-test-slave-706q krb5kdc[18244](info): set up 2 sockets
Jun 20 09:26:10 dist-test-slave-706q krb5kdc[18244](info): commencing operation
krb5kdc: starting...
W20260620 09:26:11.048213 16558 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781947562949448-16558-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781947562949448-16558-0/krb5kdc/test-user.keytab.
Jun 20 09:26:11 dist-test-slave-706q krb5kdc[18244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947571, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:26:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:26:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.43.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.43.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.43.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.190.keytab.
Entry for principal kudu/127.16.43.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.190.keytab.
I20260620 09:26:11.165899 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781947562949448-16558-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781947562949448-16558-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781947562949448-16558-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.43.190:42057
--webserver_interface=127.16.43.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.190.keytab
--principal=kudu/127.16.43.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.43.148:44019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.43.190:42057
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781947562949448-16558-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781947562949448-16558-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781947562949448-16558-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:26:11.246809 18257 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:26:11.247052 18257 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:26:11.247078 18257 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:26:11.248235 18257 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:26:11.248273 18257 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 09:26:11.248287 18257 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:26:11.248301 18257 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:26:11.248312 18257 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:26:11.248324 18257 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:26:11.249751 18257 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:44019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781947562949448-16558-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781947562949448-16558-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.43.190:42057
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.190.keytab
--principal=kudu/127.16.43.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.43.190:42057
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.43.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18257
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781947562949448-16558-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:26:11.249938 18257 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:26:11.250118 18257 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:26:11.255052 18263 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:26:11.255045 18265 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:26:11.255045 18262 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:26:11.255791 18257 server_base.cc:1034] running on GCE node
I20260620 09:26:11.256026 18257 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:26:11.256281 18257 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:26:11.257416 18257 hybrid_clock.cc:648] HybridClock initialized: now 1781947571257396 us; error 52 us; skew 500 ppm
Jun 20 09:26:11 dist-test-slave-706q krb5kdc[18244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947571, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:26:11.263116 18257 init.cc:377] Logged in from keytab as kudu/127.16.43.190@KRBTEST.COM (short username kudu)
I20260620 09:26:11.263774 18257 webserver.cc:460] Webserver started at http://127.16.43.190:39693/ using document root <none> and password file <none>
I20260620 09:26:11.264026 18257 fs_manager.cc:359] Metadata directory not provided
I20260620 09:26:11.264097 18257 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:26:11.264245 18257 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:26:11.265422 18257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781947562949448-16558-0/minicluster-data/master-0/data/instance:
uuid: "21cc679fd523463da96d6adcab8040db"
format_stamp: "Formatted at 2026-06-20 09:26:11 on dist-test-slave-706q"
server_key: "765bfd3cbabde716f48bc7bea091c84e"
server_key_iv: "68bb081adee586d190e06ba9d83fce01"
server_key_version: "encryptionkey@0"
I20260620 09:26:11.265796 18257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781947562949448-16558-0/minicluster-data/master-0/wal/instance:
uuid: "21cc679fd523463da96d6adcab8040db"
format_stamp: "Formatted at 2026-06-20 09:26:11 on dist-test-slave-706q"
server_key: "765bfd3cbabde716f48bc7bea091c84e"
server_key_iv: "68bb081adee586d190e06ba9d83fce01"
server_key_version: "encryptionkey@0"
I20260620 09:26:11.267174 18257 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 09:26:11.268103 18272 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:11.268277 18257 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:11.268411 18257 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781947562949448-16558-0/minicluster-data/master-0/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781947562949448-16558-0/minicluster-data/master-0/wal
uuid: "21cc679fd523463da96d6adcab8040db"
format_stamp: "Formatted at 2026-06-20 09:26:11 on dist-test-slave-706q"
server_key: "765bfd3cbabde716f48bc7bea091c84e"
server_key_iv: "68bb081adee586d190e06ba9d83fce01"
server_key_version: "encryptionkey@0"
I20260620 09:26:11.268496 18257 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781947562949448-16558-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781947562949448-16558-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781947562949448-16558-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:26:11.293465 18257 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:26:11.295897 18257 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:26:11.296113 18257 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:26:11.301013 18257 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.190:42057
I20260620 09:26:11.301090 18334 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.190:42057 every 8 connection(s)
I20260620 09:26:11.301411 18257 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
I20260620 09:26:11.302074 18335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:26:11.304515 18335 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21cc679fd523463da96d6adcab8040db: Bootstrap starting.
I20260620 09:26:11.305137 18335 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 21cc679fd523463da96d6adcab8040db: Neither blocks nor log segments found. Creating new log.
I20260620 09:26:11.305375 18335 log.cc:824] T 00000000000000000000000000000000 P 21cc679fd523463da96d6adcab8040db: Log is configured to *not* fsync() on all Append() calls
I20260620 09:26:11.306098 18335 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21cc679fd523463da96d6adcab8040db: No bootstrap required, opened a new log
I20260620 09:26:11.308260 18335 raft_consensus.cc:348] T 00000000000000000000000000000000 P 21cc679fd523463da96d6adcab8040db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21cc679fd523463da96d6adcab8040db" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 42057 } }
I20260620 09:26:11.308406 18335 raft_consensus.cc:374] T 00000000000000000000000000000000 P 21cc679fd523463da96d6adcab8040db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:26:11.308429 18335 raft_consensus.cc:729] T 00000000000000000000000000000000 P 21cc679fd523463da96d6adcab8040db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21cc679fd523463da96d6adcab8040db, State: Initialized, Role: FOLLOWER
I20260620 09:26:11.308583 18335 consensus_queue.cc:260] T 00000000000000000000000000000000 P 21cc679fd523463da96d6adcab8040db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21cc679fd523463da96d6adcab8040db" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 42057 } }
I20260620 09:26:11.308663 18335 raft_consensus.cc:388] T 00000000000000000000000000000000 P 21cc679fd523463da96d6adcab8040db [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:26:11.308693 18335 raft_consensus.cc:482] T 00000000000000000000000000000000 P 21cc679fd523463da96d6adcab8040db [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:26:11.308748 18335 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 21cc679fd523463da96d6adcab8040db [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:26:11.309335 18335 raft_consensus.cc:504] T 00000000000000000000000000000000 P 21cc679fd523463da96d6adcab8040db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21cc679fd523463da96d6adcab8040db" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 42057 } }
I20260620 09:26:11.309424 18335 leader_election.cc:302] T 00000000000000000000000000000000 P 21cc679fd523463da96d6adcab8040db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 21cc679fd523463da96d6adcab8040db; no voters:
I20260620 09:26:11.309607 18335 leader_election.cc:288] T 00000000000000000000000000000000 P 21cc679fd523463da96d6adcab8040db [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:26:11.309731 18338 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 21cc679fd523463da96d6adcab8040db [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:26:11.309924 18335 sys_catalog.cc:564] T 00000000000000000000000000000000 P 21cc679fd523463da96d6adcab8040db [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:26:11.309964 18338 raft_consensus.cc:686] T 00000000000000000000000000000000 P 21cc679fd523463da96d6adcab8040db [term 1 LEADER]: Becoming Leader. State: Replica: 21cc679fd523463da96d6adcab8040db, State: Running, Role: LEADER
I20260620 09:26:11.310101 18338 consensus_queue.cc:237] T 00000000000000000000000000000000 P 21cc679fd523463da96d6adcab8040db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21cc679fd523463da96d6adcab8040db" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 42057 } }
I20260620 09:26:11.310554 18339 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21cc679fd523463da96d6adcab8040db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "21cc679fd523463da96d6adcab8040db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21cc679fd523463da96d6adcab8040db" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 42057 } } }
I20260620 09:26:11.310585 18340 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21cc679fd523463da96d6adcab8040db [sys.catalog]: SysCatalogTable state changed. Reason: New leader 21cc679fd523463da96d6adcab8040db. Latest consensus state: current_term: 1 leader_uuid: "21cc679fd523463da96d6adcab8040db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21cc679fd523463da96d6adcab8040db" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 42057 } } }
I20260620 09:26:11.310654 18339 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21cc679fd523463da96d6adcab8040db [sys.catalog]: This master's current role is: LEADER
I20260620 09:26:11.310675 18340 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21cc679fd523463da96d6adcab8040db [sys.catalog]: This master's current role is: LEADER
I20260620 09:26:11.311043 18347 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:26:11.311098 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 18257
I20260620 09:26:11.311204 16558 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781947562949448-16558-0/minicluster-data/master-0/wal/instance
I20260620 09:26:11.311452 16558 external_mini_cluster.cc:1428] Setting key 5c71d7169097cd3cdea1ed948abbe264
I20260620 09:26:11.311686 18347 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:26:11.313548 18347 catalog_manager.cc:1347] Generated new cluster ID: 255f1f13ff9f491dbff831f9e0a9497b
I20260620 09:26:11.313612 18347 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 09:26:11 dist-test-slave-706q krb5kdc[18244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947571, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.43.190@KRBTEST.COM
I20260620 09:26:11.324600 18356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:11.312037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56540 (local address 127.16.43.190:42057)
0620 09:26:11.312574 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:11.312585 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:11.312624 (+ 39us) server_negotiation.cc:407] Connection header received
0620 09:26:11.312759 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:11.312766 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:11.312867 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:11.313031 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:11.313913 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:11.315316 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:11.316510 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:11.317154 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:11.317273 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:11.320345 (+ 3072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:11.320350 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:11.320350 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:26:11.320364 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:11.322681 (+ 2317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:11.323130 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:11.323133 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:11.323133 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:11.323257 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:11.323497 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:11.323497 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:11.323498 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:11.323859 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:11.324047 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:11.324155 (+ 108us) server_negotiation.cc:299] Negotiation successful
0620 09:26:11.324303 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":483,"thread_start_us":77,"threads_started":1}
I20260620 09:26:11.338529 18347 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:26:11.339290 18347 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:26:11.348224 18347 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 21cc679fd523463da96d6adcab8040db: Generated new TSK 0
I20260620 09:26:11.348467 18347 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.43.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.43.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.43.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.129.keytab.
Entry for principal kudu/127.16.43.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.129.keytab.
I20260620 09:26:11.382719 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781947562949448-16558-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781947562949448-16558-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781947562949448-16558-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781947562949448-16558-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.43.129:0
--local_ip_for_outbound_sockets=127.16.43.129
--webserver_interface=127.16.43.129
--webserver_port=0
--tserver_master_addrs=127.16.43.190:42057
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.129.keytab
--principal=kudu/127.16.43.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.43.148:44019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781947562949448-16558-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781947562949448-16558-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781947562949448-16558-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:26:11.465170 18361 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:26:11.465433 18361 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:26:11.465473 18361 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:26:11.466641 18361 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:26:11.466713 18361 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:26:11.466753 18361 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 09:26:11.466797 18361 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 09:26:11.466836 18361 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.43.129
I20260620 09:26:11.468444 18361 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:44019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781947562949448-16558-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781947562949448-16558-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.129.keytab
--principal=kudu/127.16.43.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.43.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781947562949448-16558-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.43.129
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.43.190:42057
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18361
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.43.129
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781947562949448-16558-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:26:11.468729 18361 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:26:11.468966 18361 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:26:11.474303 18367 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:26:11.474335 18369 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:26:11.474291 18366 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:26:11.474535 18361 server_base.cc:1034] running on GCE node
I20260620 09:26:11.474749 18361 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:26:11.475009 18361 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:26:11.476181 18361 hybrid_clock.cc:648] HybridClock initialized: now 1781947571476160 us; error 31 us; skew 500 ppm
Jun 20 09:26:11 dist-test-slave-706q krb5kdc[18244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947571, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:26:11.478888 18361 init.cc:377] Logged in from keytab as kudu/127.16.43.129@KRBTEST.COM (short username kudu)
I20260620 09:26:11.479286 18361 webserver.cc:460] Webserver started at http://127.16.43.129:41159/ using document root <none> and password file <none>
I20260620 09:26:11.479573 18361 fs_manager.cc:359] Metadata directory not provided
I20260620 09:26:11.479630 18361 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:26:11.479797 18361 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:26:11.480872 18361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781947562949448-16558-0/minicluster-data/ts-0/data/instance:
uuid: "9e86ec40a36f496fbb75462f500f4035"
format_stamp: "Formatted at 2026-06-20 09:26:11 on dist-test-slave-706q"
server_key: "2a937ec22c08eedbbb01bb240039b6a0"
server_key_iv: "9cb9335e26a99c45ce2dc2491b849cae"
server_key_version: "encryptionkey@0"
I20260620 09:26:11.481221 18361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781947562949448-16558-0/minicluster-data/ts-0/wal/instance:
uuid: "9e86ec40a36f496fbb75462f500f4035"
format_stamp: "Formatted at 2026-06-20 09:26:11 on dist-test-slave-706q"
server_key: "2a937ec22c08eedbbb01bb240039b6a0"
server_key_iv: "9cb9335e26a99c45ce2dc2491b849cae"
server_key_version: "encryptionkey@0"
I20260620 09:26:11.482550 18361 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 09:26:11.483294 18376 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:11.483495 18361 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 09:26:11.483632 18361 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781947562949448-16558-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781947562949448-16558-0/minicluster-data/ts-0/wal
uuid: "9e86ec40a36f496fbb75462f500f4035"
format_stamp: "Formatted at 2026-06-20 09:26:11 on dist-test-slave-706q"
server_key: "2a937ec22c08eedbbb01bb240039b6a0"
server_key_iv: "9cb9335e26a99c45ce2dc2491b849cae"
server_key_version: "encryptionkey@0"
I20260620 09:26:11.483719 18361 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781947562949448-16558-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781947562949448-16558-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781947562949448-16558-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:26:11.509097 18361 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:26:11.511721 18361 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:26:11.511881 18361 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:26:11.512446 18361 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:26:11.512483 18361 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:11.512506 18361 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:26:11.512571 18361 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:11.519613 18361 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.129:42805
I20260620 09:26:11.519709 18491 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.129:42805 every 8 connection(s)
I20260620 09:26:11.520038 18361 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781947562949448-16558-0/minicluster-data/ts-0/data/info.pb
Jun 20 09:26:11 dist-test-slave-706q krb5kdc[18244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947571, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.129@KRBTEST.COM for kudu/127.16.43.190@KRBTEST.COM
I20260620 09:26:11.523562 18356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:11.513078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.129:59037 (local address 127.16.43.190:42057)
0620 09:26:11.513136 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:11.513138 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:11.513365 (+ 227us) server_negotiation.cc:407] Connection header received
0620 09:26:11.513981 (+ 616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:11.513983 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:11.513997 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:11.514060 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:11.515070 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:11.515925 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:11.517692 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:11.517900 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:11.518009 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:11.520733 (+ 2724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:11.520734 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:11.520735 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:26:11.520746 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:11.522094 (+ 1348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:11.522530 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:11.522531 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:11.522531 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:11.522594 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:11.522871 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:11.522872 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:11.522872 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:11.523001 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:11.523080 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:11.523353 (+ 273us) server_negotiation.cc:299] Negotiation successful
0620 09:26:11.523445 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 09:26:11.523774 18385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:11.513065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:42057 (local address 127.16.43.129:59037)
0620 09:26:11.513237 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:11.513271 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:11.513798 (+ 527us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:11.514200 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:11.514205 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:11.514624 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:11.514946 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:11.514955 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:11.516084 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:11.516086 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:11.517598 (+ 1512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:11.517601 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:11.517987 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:11.517988 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:11.518158 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:11.518795 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:26:11.518808 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:26:11.520632 (+ 1824us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:26:11.522201 (+ 1569us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:11.522208 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:11.522210 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:11.522432 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:11.522663 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:11.522664 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:11.522664 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:11.522780 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:11.523070 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:26:11.523077 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:26:11.523221 (+ 144us) client_negotiation.cc:769] Sending connection context
0620 09:26:11.523332 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 09:26:11.523501 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":100,"threads_started":1}
I20260620 09:26:11.524279 18492 heartbeater.cc:347] Connected to a master server at 127.16.43.190:42057
I20260620 09:26:11.524395 18492 heartbeater.cc:464] Registering TS with master...
I20260620 09:26:11.524600 18492 heartbeater.cc:511] Master 127.16.43.190:42057 requested a full tablet report, sending...
I20260620 09:26:11.525156 18288 ts_manager.cc:194] Registered new tserver with Master: 9e86ec40a36f496fbb75462f500f4035 (127.16.43.129:42805)
I20260620 09:26:11.526018 18288 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.43.129@KRBTEST.COM'} at 127.16.43.129:59037
I20260620 09:26:11.528272 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 18361
I20260620 09:26:11.528348 16558 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781947562949448-16558-0/minicluster-data/ts-0/wal/instance
I20260620 09:26:11.528568 16558 external_mini_cluster.cc:1428] Setting key 00b954e80622c4f1912b910e2a139c8a
I20260620 09:26:11.531014 18356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:11.525419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.129:46791 (local address 127.16.43.190:42057)
0620 09:26:11.525464 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:11.525466 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:11.525581 (+ 115us) server_negotiation.cc:407] Connection header received
0620 09:26:11.525620 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:11.525621 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:11.525630 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:11.525684 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:11.525950 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:11.526519 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:11.527288 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:11.527476 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:11.527568 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:11.528483 (+ 915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:11.528485 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:11.528485 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:26:11.528492 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:11.529912 (+ 1420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:11.530308 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:11.530309 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:11.530309 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:11.530356 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:11.530558 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:11.530559 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:11.530559 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:11.530651 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:11.530736 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:11.530894 (+ 158us) server_negotiation.cc:299] Negotiation successful
0620 09:26:11.530955 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 09:26:11.531147 18498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:11.525354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:42057 (local address 127.16.43.129:46791)
0620 09:26:11.525489 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:11.525499 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:11.525571 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:11.525696 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:11.525697 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:11.525840 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:11.525888 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:11.525890 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:11.526625 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:11.526626 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:11.527218 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:11.527220 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:11.527582 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:11.527583 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:11.527659 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:11.528023 (+ 364us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:26:11.528047 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:26:11.528312 (+ 265us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:26:11.529998 (+ 1686us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:11.529998 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:11.529999 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:11.530237 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:11.530419 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:11.530420 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:11.530420 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:11.530491 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:11.530738 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:26:11.530739 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:26:11.530793 (+ 54us) client_negotiation.cc:769] Sending connection context
0620 09:26:11.530871 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 09:26:11.530987 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":85,"threads_started":1}
WARNING: no policy specified for kudu/127.16.43.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.43.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.43.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.130.keytab.
Entry for principal kudu/127.16.43.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.130.keytab.
I20260620 09:26:11.554636 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781947562949448-16558-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781947562949448-16558-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781947562949448-16558-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781947562949448-16558-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.43.130:0
--local_ip_for_outbound_sockets=127.16.43.130
--webserver_interface=127.16.43.130
--webserver_port=0
--tserver_master_addrs=127.16.43.190:42057
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.130.keytab
--principal=kudu/127.16.43.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.43.148:44019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781947562949448-16558-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781947562949448-16558-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781947562949448-16558-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:26:11.636731 18501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:26:11.636962 18501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:26:11.636989 18501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:26:11.638129 18501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:26:11.638167 18501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:26:11.638187 18501 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 09:26:11.638199 18501 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 09:26:11.638216 18501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.43.130
I20260620 09:26:11.639748 18501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:44019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781947562949448-16558-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781947562949448-16558-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.130.keytab
--principal=kudu/127.16.43.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.43.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781947562949448-16558-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.43.130
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.43.190:42057
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18501
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.43.130
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781947562949448-16558-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:26:11.639959 18501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:26:11.640137 18501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:26:11.645438 18509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:26:11.645439 18507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:26:11.645674 18501 server_base.cc:1034] running on GCE node
W20260620 09:26:11.645505 18506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:26:11.645977 18501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:26:11.646266 18501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:26:11.647425 18501 hybrid_clock.cc:648] HybridClock initialized: now 1781947571647407 us; error 45 us; skew 500 ppm
Jun 20 09:26:11 dist-test-slave-706q krb5kdc[18244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947571, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:26:11.650074 18501 init.cc:377] Logged in from keytab as kudu/127.16.43.130@KRBTEST.COM (short username kudu)
I20260620 09:26:11.650524 18501 webserver.cc:460] Webserver started at http://127.16.43.130:33207/ using document root <none> and password file <none>
I20260620 09:26:11.650748 18501 fs_manager.cc:359] Metadata directory not provided
I20260620 09:26:11.650796 18501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:26:11.650924 18501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:26:11.652086 18501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781947562949448-16558-0/minicluster-data/ts-1/data/instance:
uuid: "41789aa446ae4c5bbcda631441a01566"
format_stamp: "Formatted at 2026-06-20 09:26:11 on dist-test-slave-706q"
server_key: "a5d155db8a9927c810743ac3534932df"
server_key_iv: "39226e7b22eb0154b2e8882f447870be"
server_key_version: "encryptionkey@0"
I20260620 09:26:11.652456 18501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781947562949448-16558-0/minicluster-data/ts-1/wal/instance:
uuid: "41789aa446ae4c5bbcda631441a01566"
format_stamp: "Formatted at 2026-06-20 09:26:11 on dist-test-slave-706q"
server_key: "a5d155db8a9927c810743ac3534932df"
server_key_iv: "39226e7b22eb0154b2e8882f447870be"
server_key_version: "encryptionkey@0"
I20260620 09:26:11.653797 18501 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260620 09:26:11.654590 18516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:11.654810 18501 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:26:11.654913 18501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781947562949448-16558-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781947562949448-16558-0/minicluster-data/ts-1/wal
uuid: "41789aa446ae4c5bbcda631441a01566"
format_stamp: "Formatted at 2026-06-20 09:26:11 on dist-test-slave-706q"
server_key: "a5d155db8a9927c810743ac3534932df"
server_key_iv: "39226e7b22eb0154b2e8882f447870be"
server_key_version: "encryptionkey@0"
I20260620 09:26:11.654994 18501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781947562949448-16558-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781947562949448-16558-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781947562949448-16558-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:26:11.669188 18501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:26:11.671777 18501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:26:11.671898 18501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:26:11.672472 18501 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:26:11.672518 18501 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:11.672549 18501 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:26:11.672560 18501 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:11.679800 18501 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.130:41977
I20260620 09:26:11.679900 18631 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.130:41977 every 8 connection(s)
I20260620 09:26:11.680204 18501 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781947562949448-16558-0/minicluster-data/ts-1/data/info.pb
Jun 20 09:26:11 dist-test-slave-706q krb5kdc[18244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947571, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.130@KRBTEST.COM for kudu/127.16.43.190@KRBTEST.COM
I20260620 09:26:11.684054 18356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:11.673032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.130:43877 (local address 127.16.43.190:42057)
0620 09:26:11.673084 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:11.673085 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:11.673370 (+ 285us) server_negotiation.cc:407] Connection header received
0620 09:26:11.673981 (+ 611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:11.673983 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:11.673996 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:11.674067 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:11.675015 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:11.675906 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:11.677655 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:11.677864 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:11.677932 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:11.680966 (+ 3034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:11.680967 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:11.680968 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:26:11.680974 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:11.682315 (+ 1341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:11.682743 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:11.682744 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:11.682744 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:11.682806 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:11.683215 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:11.683216 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:11.683216 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:11.683340 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:11.683400 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:11.683830 (+ 430us) server_negotiation.cc:299] Negotiation successful
0620 09:26:11.683978 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 09:26:11.684208 18525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:11.673049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:42057 (local address 127.16.43.130:43877)
0620 09:26:11.673241 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:11.673279 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:11.673871 (+ 592us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:11.674190 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:11.674195 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:11.674604 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:11.674920 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:11.674928 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:11.676034 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:11.676044 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:11.677552 (+ 1508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:11.677556 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:11.677969 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:11.677970 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:11.678194 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:11.678881 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:26:11.678898 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:26:11.680856 (+ 1958us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:26:11.682405 (+ 1549us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:11.682412 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:11.682414 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:11.682641 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:11.682979 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:11.682980 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:11.682980 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:11.683117 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:11.683418 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:26:11.683425 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:26:11.683649 (+ 224us) client_negotiation.cc:769] Sending connection context
0620 09:26:11.683760 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 09:26:11.683971 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":104,"threads_started":1}
I20260620 09:26:11.684794 18632 heartbeater.cc:347] Connected to a master server at 127.16.43.190:42057
I20260620 09:26:11.684973 18632 heartbeater.cc:464] Registering TS with master...
I20260620 09:26:11.685173 18632 heartbeater.cc:511] Master 127.16.43.190:42057 requested a full tablet report, sending...
I20260620 09:26:11.685617 18289 ts_manager.cc:194] Registered new tserver with Master: 41789aa446ae4c5bbcda631441a01566 (127.16.43.130:41977)
I20260620 09:26:11.686164 18289 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.43.130@KRBTEST.COM'} at 127.16.43.130:43877
I20260620 09:26:11.689962 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 18501
I20260620 09:26:11.690085 16558 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781947562949448-16558-0/minicluster-data/ts-1/wal/instance
I20260620 09:26:11.690326 16558 external_mini_cluster.cc:1428] Setting key 8ffb7ff1a0b30de23a5e10e9796318f5
I20260620 09:26:11.692209 18356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:11.686120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.130:44799 (local address 127.16.43.190:42057)
0620 09:26:11.686168 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:11.686170 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:11.686386 (+ 216us) server_negotiation.cc:407] Connection header received
0620 09:26:11.686494 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:11.686495 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:11.686506 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:11.686563 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:11.686852 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:11.687412 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:11.688172 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:11.688348 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:11.688400 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:11.689366 (+ 966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:11.689367 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:11.689367 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:26:11.689373 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:11.690918 (+ 1545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:11.691310 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:11.691311 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:11.691312 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:11.691371 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:11.691642 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:11.691643 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:11.691643 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:11.691799 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:11.691881 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:11.692037 (+ 156us) server_negotiation.cc:299] Negotiation successful
0620 09:26:11.692116 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 09:26:11.692219 18638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:11.686019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:42057 (local address 127.16.43.130:44799)
0620 09:26:11.686325 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:11.686336 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:11.686400 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:11.686581 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:11.686581 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:11.686735 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:11.686784 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:11.686785 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:11.687559 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:11.687560 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:11.688099 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:11.688101 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:11.688426 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:11.688427 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:11.688484 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:11.688983 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:26:11.688988 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:26:11.689288 (+ 300us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:26:11.691011 (+ 1723us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:11.691013 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:11.691013 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:11.691217 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:11.691470 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:11.691471 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:11.691471 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:11.691578 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:11.691871 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:26:11.691872 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:26:11.691922 (+ 50us) client_negotiation.cc:769] Sending connection context
0620 09:26:11.692012 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 09:26:11.692117 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":64,"threads_started":1}
WARNING: no policy specified for kudu/127.16.43.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.43.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.43.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.131.keytab.
Entry for principal kudu/127.16.43.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.131.keytab.
I20260620 09:26:11.714540 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781947562949448-16558-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781947562949448-16558-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781947562949448-16558-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781947562949448-16558-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.43.131:0
--local_ip_for_outbound_sockets=127.16.43.131
--webserver_interface=127.16.43.131
--webserver_port=0
--tserver_master_addrs=127.16.43.190:42057
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.131.keytab
--principal=kudu/127.16.43.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.43.148:44019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781947562949448-16558-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781947562949448-16558-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781947562949448-16558-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:26:11.796792 18641 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:26:11.797005 18641 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:26:11.797030 18641 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:26:11.798218 18641 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:26:11.798260 18641 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:26:11.798282 18641 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 09:26:11.798295 18641 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 09:26:11.798311 18641 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.43.131
I20260620 09:26:11.800347 18641 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:44019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781947562949448-16558-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781947562949448-16558-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.131.keytab
--principal=kudu/127.16.43.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.43.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781947562949448-16558-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.43.131
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.43.190:42057
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18641
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.43.131
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781947562949448-16558-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:26:11.800602 18641 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:26:11.800853 18641 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:26:11.806113 18647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:26:11.806164 18641 server_base.cc:1034] running on GCE node
W20260620 09:26:11.806120 18646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:26:11.806133 18649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:26:11.806494 18641 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:26:11.806773 18641 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:26:11.807971 18641 hybrid_clock.cc:648] HybridClock initialized: now 1781947571807956 us; error 30 us; skew 500 ppm
Jun 20 09:26:11 dist-test-slave-706q krb5kdc[18244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947571, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:26:11.810976 18641 init.cc:377] Logged in from keytab as kudu/127.16.43.131@KRBTEST.COM (short username kudu)
I20260620 09:26:11.811615 18641 webserver.cc:460] Webserver started at http://127.16.43.131:34941/ using document root <none> and password file <none>
I20260620 09:26:11.811823 18641 fs_manager.cc:359] Metadata directory not provided
I20260620 09:26:11.811867 18641 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:26:11.811962 18641 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:26:11.813031 18641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781947562949448-16558-0/minicluster-data/ts-2/data/instance:
uuid: "b9807e3fbedb4973a065b0c81113fa6a"
format_stamp: "Formatted at 2026-06-20 09:26:11 on dist-test-slave-706q"
server_key: "284301a50a9b3db17c0cdc7f7f9e47da"
server_key_iv: "3bcc52c6eaae7d841df3d0def9c36a8c"
server_key_version: "encryptionkey@0"
I20260620 09:26:11.813328 18641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781947562949448-16558-0/minicluster-data/ts-2/wal/instance:
uuid: "b9807e3fbedb4973a065b0c81113fa6a"
format_stamp: "Formatted at 2026-06-20 09:26:11 on dist-test-slave-706q"
server_key: "284301a50a9b3db17c0cdc7f7f9e47da"
server_key_iv: "3bcc52c6eaae7d841df3d0def9c36a8c"
server_key_version: "encryptionkey@0"
I20260620 09:26:11.814615 18641 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 09:26:11.815284 18656 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:11.815595 18641 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 09:26:11.815685 18641 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781947562949448-16558-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781947562949448-16558-0/minicluster-data/ts-2/wal
uuid: "b9807e3fbedb4973a065b0c81113fa6a"
format_stamp: "Formatted at 2026-06-20 09:26:11 on dist-test-slave-706q"
server_key: "284301a50a9b3db17c0cdc7f7f9e47da"
server_key_iv: "3bcc52c6eaae7d841df3d0def9c36a8c"
server_key_version: "encryptionkey@0"
I20260620 09:26:11.815739 18641 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781947562949448-16558-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781947562949448-16558-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781947562949448-16558-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:26:11.830412 18641 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:26:11.832561 18641 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:26:11.832672 18641 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:26:11.833209 18641 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:26:11.833245 18641 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:11.833269 18641 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:26:11.833281 18641 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:11.840591 18641 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.131:41849
I20260620 09:26:11.840660 18771 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.131:41849 every 8 connection(s)
I20260620 09:26:11.840998 18641 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781947562949448-16558-0/minicluster-data/ts-2/data/info.pb
Jun 20 09:26:11 dist-test-slave-706q krb5kdc[18244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947571, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.131@KRBTEST.COM for kudu/127.16.43.190@KRBTEST.COM
I20260620 09:26:11.844408 18356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:11.833826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.131:45981 (local address 127.16.43.190:42057)
0620 09:26:11.833878 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:11.833880 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:11.834146 (+ 266us) server_negotiation.cc:407] Connection header received
0620 09:26:11.834732 (+ 586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:11.834734 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:11.834748 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:11.834833 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:11.835901 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:11.836824 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:11.838547 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:11.838735 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:11.838798 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:11.841666 (+ 2868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:11.841667 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:11.841668 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:26:11.841674 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:11.843165 (+ 1491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:11.843633 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:11.843634 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:11.843634 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:11.843686 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:11.843912 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:11.843912 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:11.843912 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:11.844017 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:11.844076 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:11.844284 (+ 208us) server_negotiation.cc:299] Negotiation successful
0620 09:26:11.844344 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 09:26:11.844630 18665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:11.833835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:42057 (local address 127.16.43.131:45981)
0620 09:26:11.834021 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:11.834057 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:11.834613 (+ 556us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:11.834973 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:11.834979 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:11.835415 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:11.835773 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:11.835792 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:11.836934 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:11.836935 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:11.838445 (+ 1510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:11.838448 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:11.838830 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:11.838832 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:11.839004 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:11.839670 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:26:11.839681 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:26:11.841564 (+ 1883us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:26:11.843244 (+ 1680us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:11.843250 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:11.843252 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:11.843489 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:11.843745 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:11.843746 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:11.843746 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:11.843848 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:11.844071 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:26:11.844075 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:26:11.844188 (+ 113us) client_negotiation.cc:769] Sending connection context
0620 09:26:11.844265 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 09:26:11.844383 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":88,"threads_started":1}
I20260620 09:26:11.845204 18772 heartbeater.cc:347] Connected to a master server at 127.16.43.190:42057
I20260620 09:26:11.845299 18772 heartbeater.cc:464] Registering TS with master...
I20260620 09:26:11.845486 18772 heartbeater.cc:511] Master 127.16.43.190:42057 requested a full tablet report, sending...
I20260620 09:26:11.845875 18288 ts_manager.cc:194] Registered new tserver with Master: b9807e3fbedb4973a065b0c81113fa6a (127.16.43.131:41849)
I20260620 09:26:11.846312 18288 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.43.131@KRBTEST.COM'} at 127.16.43.131:45981
I20260620 09:26:11.849694 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 18641
I20260620 09:26:11.849793 16558 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781947562949448-16558-0/minicluster-data/ts-2/wal/instance
I20260620 09:26:11.849957 16558 external_mini_cluster.cc:1428] Setting key 02692b8f20b1179b5626f65555b46df0
I20260620 09:26:11.850863 16558 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 09:26:11.852613 18356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:11.846353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.131:56729 (local address 127.16.43.190:42057)
0620 09:26:11.846421 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:11.846423 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:11.846489 (+ 66us) server_negotiation.cc:407] Connection header received
0620 09:26:11.846558 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:11.846559 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:11.846568 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:11.846616 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:11.847433 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:11.847996 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:11.848776 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:11.848975 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:11.849057 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:11.849908 (+ 851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:11.849910 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:11.849910 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:26:11.849917 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:11.851433 (+ 1516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:11.851820 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:11.851821 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:11.851822 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:11.851877 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:11.852089 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:11.852089 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:11.852090 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:11.852203 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:11.852279 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:11.852445 (+ 166us) server_negotiation.cc:299] Negotiation successful
0620 09:26:11.852520 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 09:26:11.852628 18778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:11.846284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:42057 (local address 127.16.43.131:56729)
0620 09:26:11.846434 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:11.846445 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:11.846503 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:11.847129 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:11.847130 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:11.847274 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:11.847326 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:11.847327 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:11.848102 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:11.848103 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:11.848675 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:11.848677 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:11.849060 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:11.849061 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:11.849138 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:11.849490 (+ 352us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:26:11.849494 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:26:11.849781 (+ 287us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:26:11.851518 (+ 1737us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:11.851519 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:11.851520 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:11.851733 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:11.851955 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:11.851956 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:11.851956 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:11.852022 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:11.852270 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:26:11.852271 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:26:11.852329 (+ 58us) client_negotiation.cc:769] Sending connection context
0620 09:26:11.852418 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 09:26:11.852523 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":84,"threads_started":1}
Jun 20 09:26:11 dist-test-slave-706q krb5kdc[18244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947571, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 09:26:11 dist-test-slave-706q krb5kdc[18244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947571, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.43.190@KRBTEST.COM
I20260620 09:26:11.875268 18356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:11.867347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56554 (local address 127.16.43.190:42057)
0620 09:26:11.867400 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:11.867402 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:11.867449 (+ 47us) server_negotiation.cc:407] Connection header received
0620 09:26:11.867597 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:11.867598 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:11.867610 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:11.867669 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:11.868403 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:11.868891 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:11.869713 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:11.869902 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:11.869963 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:11.872716 (+ 2753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:11.872717 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:11.872717 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:26:11.872723 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:11.874093 (+ 1370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:11.874590 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:11.874591 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:11.874591 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:11.874645 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:11.874834 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:11.874835 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:11.874835 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:11.874947 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:11.875022 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:11.875140 (+ 118us) server_negotiation.cc:299] Negotiation successful
0620 09:26:11.875199 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 09:26:11.877211 18288 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56554:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 09:26:11.878798 18288 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 09:26:11.886642 18793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:11.881615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.129:42805 (local address 127.0.0.1:53164)
0620 09:26:11.882057 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:11.882069 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:11.882144 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:11.882565 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:11.882567 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:11.882569 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:26:11.882634 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:11.882636 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:11.883882 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:11.883883 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:11.885225 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:11.885228 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:11.886282 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:11.886283 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:11.886379 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:11.886397 (+ 18us) client_negotiation.cc:769] Sending connection context
0620 09:26:11.886481 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 09:26:11.886541 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":414,"mutex_wait_us":189,"thread_start_us":90,"threads_started":1}
I20260620 09:26:11.886929 18789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:11.881842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.131:41849 (local address 127.0.0.1:52330)
0620 09:26:11.882387 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:11.882398 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:11.882451 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:11.882724 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:11.882724 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:11.882726 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:26:11.882775 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:11.882776 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:11.883805 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:11.883806 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:11.885526 (+ 1720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:11.885528 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:11.886679 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:11.886680 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:11.886746 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:11.886751 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 09:26:11.886807 (+ 56us) client_negotiation.cc:240] Negotiation successful
0620 09:26:11.886851 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"mutex_wait_us":139,"thread_start_us":170,"threads_started":1}
I20260620 09:26:11.887120 18791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:11.881711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53164 (local address 127.16.43.129:42805)
0620 09:26:11.881903 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:11.881909 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:11.882255 (+ 346us) server_negotiation.cc:407] Connection header received
0620 09:26:11.882381 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:11.882383 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:11.882404 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:11.882489 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:26:11.882718 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:11.883819 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:11.885319 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:11.886191 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:11.886268 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:11.886329 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:11.886820 (+ 491us) server_negotiation.cc:299] Negotiation successful
0620 09:26:11.886909 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":93,"threads_started":1}
I20260620 09:26:11.887274 18794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:11.882156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52330 (local address 127.16.43.131:41849)
0620 09:26:11.882343 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:11.882346 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:11.882515 (+ 169us) server_negotiation.cc:407] Connection header received
0620 09:26:11.882557 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:11.882559 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:11.882574 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:11.882648 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:26:11.882857 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:11.883711 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:11.885628 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:11.886522 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:11.886587 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:11.886642 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:11.887111 (+ 469us) server_negotiation.cc:299] Negotiation successful
0620 09:26:11.887163 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":81,"threads_started":1}
I20260620 09:26:11.887459 18790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:11.881599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.130:41977 (local address 127.0.0.1:53952)
0620 09:26:11.881983 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:11.882017 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:11.882117 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:11.882372 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:11.882375 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:11.882378 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:26:11.882450 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:11.882452 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:11.883862 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:11.883863 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:11.885505 (+ 1642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:11.885509 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:11.886632 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:11.886633 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:11.886704 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:11.886708 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 09:26:11.887183 (+ 475us) client_negotiation.cc:240] Negotiation successful
0620 09:26:11.887306 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"mutex_wait_us":188,"thread_start_us":73,"threads_started":1}
I20260620 09:26:11.887482 18792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:11.881765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53952 (local address 127.16.43.130:41977)
0620 09:26:11.881963 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:11.881969 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:11.882097 (+ 128us) server_negotiation.cc:407] Connection header received
0620 09:26:11.882202 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:11.882204 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:11.882219 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:11.882293 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:26:11.882546 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:11.883763 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:11.885602 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:11.886381 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:11.886478 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:11.886538 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:11.887270 (+ 732us) server_negotiation.cc:299] Negotiation successful
0620 09:26:11.887384 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":114,"threads_started":1}
I20260620 09:26:11.888015 18706 tablet_service.cc:1467] Processing CreateTablet for tablet 6a2b6a5f6d2e42c3be7e0bf4c4ce5f3f (DEFAULT_TABLE table=test-table [id=3517409efffb41508f4b8e4245eb00bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:26:11.888202 18566 tablet_service.cc:1467] Processing CreateTablet for tablet 6a2b6a5f6d2e42c3be7e0bf4c4ce5f3f (DEFAULT_TABLE table=test-table [id=3517409efffb41508f4b8e4245eb00bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:26:11.888422 18706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a2b6a5f6d2e42c3be7e0bf4c4ce5f3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:26:11.888573 18566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a2b6a5f6d2e42c3be7e0bf4c4ce5f3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:26:11.891366 18795 tablet_bootstrap.cc:492] T 6a2b6a5f6d2e42c3be7e0bf4c4ce5f3f P b9807e3fbedb4973a065b0c81113fa6a: Bootstrap starting.
I20260620 09:26:11.891763 18796 tablet_bootstrap.cc:492] T 6a2b6a5f6d2e42c3be7e0bf4c4ce5f3f P 41789aa446ae4c5bbcda631441a01566: Bootstrap starting.
I20260620 09:26:11.892060 18426 tablet_service.cc:1467] Processing CreateTablet for tablet 6a2b6a5f6d2e42c3be7e0bf4c4ce5f3f (DEFAULT_TABLE table=test-table [id=3517409efffb41508f4b8e4245eb00bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:26:11.892379 18796 tablet_bootstrap.cc:654] T 6a2b6a5f6d2e42c3be7e0bf4c4ce5f3f P 41789aa446ae4c5bbcda631441a01566: Neither blocks nor log segments found. Creating new log.
I20260620 09:26:11.892477 18426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a2b6a5f6d2e42c3be7e0bf4c4ce5f3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:26:11.892516 18795 tablet_bootstrap.cc:654] T 6a2b6a5f6d2e42c3be7e0bf4c4ce5f3f P b9807e3fbedb4973a065b0c81113fa6a: Neither blocks nor log segments found. Creating new log.
I20260620 09:26:11.892714 18796 log.cc:824] T 6a2b6a5f6d2e42c3be7e0bf4c4ce5f3f P 41789aa446ae4c5bbcda631441a01566: Log is configured to *not* fsync() on all Append() calls
I20260620 09:26:11.892853 18795 log.cc:824] T 6a2b6a5f6d2e42c3be7e0bf4c4ce5f3f P b9807e3fbedb4973a065b0c81113fa6a: Log is configured to *not* fsync() on all Append() calls
I20260620 09:26:11.893608 18795 tablet_bootstrap.cc:492] T 6a2b6a5f6d2e42c3be7e0bf4c4ce5f3f P b9807e3fbedb4973a065b0c81113fa6a: No bootstrap required, opened a new log
I20260620 09:26:11.893615 18796 tablet_bootstrap.cc:492] T 6a2b6a5f6d2e42c3be7e0bf4c4ce5f3f P 41789aa446ae4c5bbcda631441a01566: No bootstrap required, opened a new log
I20260620 09:26:11.893683 18796 ts_tablet_manager.cc:1397] T 6a2b6a5f6d2e42c3be7e0bf4c4ce5f3f P 41789aa446ae4c5bbcda631441a01566: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 09:26:11.893683 18795 ts_tablet_manager.cc:1397] T 6a2b6a5f6d2e42c3be7e0bf4c4ce5f3f P b9807e3fbedb4973a065b0c81113fa6a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 09:26:11.895738 18796 raft_consensus.cc:348] T 6a2b6a5f6d2e42c3be7e0bf4c4ce5f3f P 41789aa446ae4c5bbcda631441a01566 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e86ec40a36f496fbb75462f500f4035" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 42805 } } peers { permanent_uuid: "41789aa446ae4c5bbcda631441a01566" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 41977 } } peers { permanent_uuid: "b9807e3fbedb4973a065b0c81113fa6a" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 41849 } }
I20260620 09:26:11.895738 18795 raft_consensus.cc:348] T 6a2b6a5f6d2e42c3be7e0bf4c4ce5f3f P b9807e3fbedb4973a065b0c81113fa6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e86ec40a36f496fbb75462f500f4035" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 42805 } } peers { permanent_uuid: "41789aa446ae4c5bbcda631441a01566" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 41977 } } peers { permanent_uuid: "b9807e3fbedb4973a065b0c81113fa6a" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 41849 } }
I20260620 09:26:11.895911 18795 raft_consensus.cc:374] T 6a2b6a5f6d2e42c3be7e0bf4c4ce5f3f P b9807e3fbedb4973a065b0c81113fa6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:26:11.895911 18796 raft_consensus.cc:374] T 6a2b6a5f6d2e42c3be7e0bf4c4ce5f3f P 41789aa446ae4c5bbcda631441a01566 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:26:11.895955 18795 raft_consensus.cc:729] T 6a2b6a5f6d2e42c3be7e0bf4c4ce5f3f P b9807e3fbedb4973a065b0c81113fa6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9807e3fbedb4973a065b0c81113fa6a, State: Initialized, Role: FOLLOWER
I20260620 09:26:11.895955 18796 raft_consensus.cc:729] T 6a2b6a5f6d2e42c3be7e0bf4c4ce5f3f P 41789aa446ae4c5bbcda631441a01566 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41789aa446ae4c5bbcda631441a01566, State: Initialized, Role: FOLLOWER
I20260620 09:26:11.895835 18799 tablet_bootstrap.cc:492] T 6a2b6a5f6d2e42c3be7e0bf4c4ce5f3f P 9e86ec40a36f496fbb75462f500f4035: Bootstrap starting.
I20260620 09:26:11.896082 18795 consensus_queue.cc:260] T 6a2b6a5f6d2e42c3be7e0bf4c4ce5f3f P b9807e3fbedb4973a065b0c81113fa6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e86ec40a36f496fbb75462f500f4035" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 42805 } } peers { permanent_uuid: "41789aa446ae4c5bbcda631441a01566" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 41977 } } peers { permanent_uuid: "b9807e3fbedb4973a065b0c81113fa6a" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 41849 } }
I20260620 09:26:11.896075 18796 consensus_queue.cc:260] T 6a2b6a5f6d2e42c3be7e0bf4c4ce5f3f P 41789aa446ae4c5bbcda631441a01566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e86ec40a36f496fbb75462f500f4035" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 42805 } } peers { permanent_uuid: "41789aa446ae4c5bbcda631441a01566" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 41977 } } peers { permanent_uuid: "b9807e3fbedb4973a065b0c81113fa6a" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 41849 } }
I20260620 09:26:11.896343 18796 ts_tablet_manager.cc:1428] T 6a2b6a5f6d2e42c3be7e0bf4c4ce5f3f P 41789aa446ae4c5bbcda631441a01566: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260620 09:26:11.896343 18795 ts_tablet_manager.cc:1428] T 6a2b6a5f6d2e42c3be7e0bf4c4ce5f3f P b9807e3fbedb4973a065b0c81113fa6a: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260620 09:26:11.896461 18772 heartbeater.cc:503] Master 127.16.43.190:42057 was elected leader, sending a full tablet report...
I20260620 09:26:11.896540 18632 heartbeater.cc:503] Master 127.16.43.190:42057 was elected leader, sending a full tablet report...
I20260620 09:26:11.896757 18799 tablet_bootstrap.cc:654] T 6a2b6a5f6d2e42c3be7e0bf4c4ce5f3f P 9e86ec40a36f496fbb75462f500f4035: Neither blocks nor log segments found. Creating new log.
I20260620 09:26:11.897142 18799 log.cc:824] T 6a2b6a5f6d2e42c3be7e0bf4c4ce5f3f P 9e86ec40a36f496fbb75462f500f4035: Log is configured to *not* fsync() on all Append() calls
I20260620 09:26:11.898030 18799 tablet_bootstrap.cc:492] T 6a2b6a5f6d2e42c3be7e0bf4c4ce5f3f P 9e86ec40a36f496fbb75462f500f4035: No bootstrap required, opened a new log
I20260620 09:26:11.898088 18799 ts_tablet_manager.cc:1397] T 6a2b6a5f6d2e42c3be7e0bf4c4ce5f3f P 9e86ec40a36f496fbb75462f500f4035: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 09:26:11.899477 18799 raft_consensus.cc:348] T 6a2b6a5f6d2e42c3be7e0bf4c4ce5f3f P 9e86ec40a36f496fbb75462f500f4035 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e86ec40a36f496fbb75462f500f4035" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 42805 } } peers { permanent_uuid: "41789aa446ae4c5bbcda631441a01566" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 41977 } } peers { permanent_uuid: "b9807e3fbedb4973a065b0c81113fa6a" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 41849 } }
I20260620 09:26:11.899698 18799 raft_consensus.cc:374] T 6a2b6a5f6d2e42c3be7e0bf4c4ce5f3f P 9e86ec40a36f496fbb75462f500f4035 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:26:11.899750 18799 raft_consensus.cc:729] T 6a2b6a5f6d2e42c3be7e0bf4c4ce5f3f P 9e86ec40a36f496fbb75462f500f4035 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e86ec40a36f496fbb75462f500f4035, State: Initialized, Role: FOLLOWER
I20260620 09:26:11.899844 18799 consensus_queue.cc:260] T 6a2b6a5f6d2e42c3be7e0bf4c4ce5f3f P 9e86ec40a36f496fbb75462f500f4035 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e86ec40a36f496fbb75462f500f4035" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 42805 } } peers { permanent_uuid: "41789aa446ae4c5bbcda631441a01566" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 41977 } } peers { permanent_uuid: "b9807e3fbedb4973a065b0c81113fa6a" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 41849 } }
I20260620 09:26:11.900060 18799 ts_tablet_manager.cc:1428] T 6a2b6a5f6d2e42c3be7e0bf4c4ce5f3f P 9e86ec40a36f496fbb75462f500f4035: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 09:26:11.900130 18492 heartbeater.cc:503] Master 127.16.43.190:42057 was elected leader, sending a full tablet report...
W20260620 09:26:11.930922 18633 tablet.cc:2369] T 6a2b6a5f6d2e42c3be7e0bf4c4ce5f3f P 41789aa446ae4c5bbcda631441a01566: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 09:26:11.937433 18801 raft_consensus.cc:482] T 6a2b6a5f6d2e42c3be7e0bf4c4ce5f3f P 41789aa446ae4c5bbcda631441a01566 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:26:11.937572 18801 raft_consensus.cc:504] T 6a2b6a5f6d2e42c3be7e0bf4c4ce5f3f P 41789aa446ae4c5bbcda631441a01566 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e86ec40a36f496fbb75462f500f4035" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 42805 } } peers { permanent_uuid: "41789aa446ae4c5bbcda631441a01566" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 41977 } } peers { permanent_uuid: "b9807e3fbedb4973a065b0c81113fa6a" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 41849 } }
I20260620 09:26:11.938107 18801 leader_election.cc:288] T 6a2b6a5f6d2e42c3be7e0bf4c4ce5f3f P 41789aa446ae4c5bbcda631441a01566 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9e86ec40a36f496fbb75462f500f4035 (127.16.43.129:42805), b9807e3fbedb4973a065b0c81113fa6a (127.16.43.131:41849)
I20260620 09:26:11.941612 18794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:11.938437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.130:43555 (local address 127.16.43.131:41849)
0620 09:26:11.938493 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:11.938496 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:11.938611 (+ 115us) server_negotiation.cc:407] Connection header received
0620 09:26:11.938702 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:11.938703 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:11.938710 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:11.938754 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:26:11.938952 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:11.939665 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:11.940786 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:11.941178 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:11.941330 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:11.941340 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:11.941503 (+ 163us) server_negotiation.cc:299] Negotiation successful
0620 09:26:11.941544 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 09:26:11.941627 18804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:11.938333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.131:41849 (local address 127.16.43.130:43555)
0620 09:26:11.938524 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:11.938534 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:11.938623 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:11.938781 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:11.938782 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:11.938784 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:26:11.938878 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:11.938879 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:11.939756 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:11.939757 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:11.940690 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:11.940693 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:11.941339 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:11.941340 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:11.941420 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:11.941423 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 09:26:11.941486 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 09:26:11.941527 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"mutex_wait_us":1,"thread_start_us":121,"threads_started":1}
I20260620 09:26:11.941716 18525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:11.938313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.129:42805 (local address 127.16.43.130:43037)
0620 09:26:11.938377 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:11.938390 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:11.938452 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:11.938636 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:11.938637 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:11.938639 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:26:11.938719 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:11.938721 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:11.939731 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:11.939732 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:11.940704 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:11.940707 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:11.941417 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:11.941418 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:11.941516 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:11.941519 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 09:26:11.941588 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 09:26:11.941627 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":21}
I20260620 09:26:11.941707 18791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:11.938369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.130:43037 (local address 127.16.43.129:42805)
0620 09:26:11.938420 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:11.938423 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:11.938439 (+ 16us) server_negotiation.cc:407] Connection header received
0620 09:26:11.938524 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:11.938525 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:11.938537 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:11.938627 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:26:11.938832 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:11.939635 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:11.940821 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:11.941315 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:11.941406 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:11.941415 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:11.941594 (+ 179us) server_negotiation.cc:299] Negotiation successful
0620 09:26:11.941636 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 09:26:11.942008 18446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a2b6a5f6d2e42c3be7e0bf4c4ce5f3f" candidate_uuid: "41789aa446ae4c5bbcda631441a01566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e86ec40a36f496fbb75462f500f4035" is_pre_election: true
I20260620 09:26:11.941984 18726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a2b6a5f6d2e42c3be7e0bf4c4ce5f3f" candidate_uuid: "41789aa446ae4c5bbcda631441a01566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9807e3fbedb4973a065b0c81113fa6a" is_pre_election: true
I20260620 09:26:11.942158 18446 raft_consensus.cc:2454] T 6a2b6a5f6d2e42c3be7e0bf4c4ce5f3f P 9e86ec40a36f496fbb75462f500f4035 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41789aa446ae4c5bbcda631441a01566 in term 0.
I20260620 09:26:11.942158 18726 raft_consensus.cc:2454] T 6a2b6a5f6d2e42c3be7e0bf4c4ce5f3f P b9807e3fbedb4973a065b0c81113fa6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41789aa446ae4c5bbcda631441a01566 in term 0.
I20260620 09:26:11.942401 18519 leader_election.cc:302] T 6a2b6a5f6d2e42c3be7e0bf4c4ce5f3f P 41789aa446ae4c5bbcda631441a01566 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41789aa446ae4c5bbcda631441a01566, b9807e3fbedb4973a065b0c81113fa6a; no voters:
I20260620 09:26:11.942554 18801 raft_consensus.cc:2781] T 6a2b6a5f6d2e42c3be7e0bf4c4ce5f3f P 41789aa446ae4c5bbcda631441a01566 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 09:26:11.942627 18801 raft_consensus.cc:482] T 6a2b6a5f6d2e42c3be7e0bf4c4ce5f3f P 41789aa446ae4c5bbcda631441a01566 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 09:26:11.942646 18801 raft_consensus.cc:3037] T 6a2b6a5f6d2e42c3be7e0bf4c4ce5f3f P 41789aa446ae4c5bbcda631441a01566 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:26:11.943631 18801 raft_consensus.cc:504] T 6a2b6a5f6d2e42c3be7e0bf4c4ce5f3f P 41789aa446ae4c5bbcda631441a01566 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e86ec40a36f496fbb75462f500f4035" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 42805 } } peers { permanent_uuid: "41789aa446ae4c5bbcda631441a01566" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 41977 } } peers { permanent_uuid: "b9807e3fbedb4973a065b0c81113fa6a" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 41849 } }
I20260620 09:26:11.943785 18801 leader_election.cc:288] T 6a2b6a5f6d2e42c3be7e0bf4c4ce5f3f P 41789aa446ae4c5bbcda631441a01566 [CANDIDATE]: Term 1 election: Requested vote from peers 9e86ec40a36f496fbb75462f500f4035 (127.16.43.129:42805), b9807e3fbedb4973a065b0c81113fa6a (127.16.43.131:41849)
I20260620 09:26:11.943951 18726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a2b6a5f6d2e42c3be7e0bf4c4ce5f3f" candidate_uuid: "41789aa446ae4c5bbcda631441a01566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9807e3fbedb4973a065b0c81113fa6a"
I20260620 09:26:11.943972 18446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a2b6a5f6d2e42c3be7e0bf4c4ce5f3f" candidate_uuid: "41789aa446ae4c5bbcda631441a01566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e86ec40a36f496fbb75462f500f4035"
I20260620 09:26:11.944062 18446 raft_consensus.cc:3037] T 6a2b6a5f6d2e42c3be7e0bf4c4ce5f3f P 9e86ec40a36f496fbb75462f500f4035 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:26:11.944062 18726 raft_consensus.cc:3037] T 6a2b6a5f6d2e42c3be7e0bf4c4ce5f3f P b9807e3fbedb4973a065b0c81113fa6a [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:26:11.944983 18446 raft_consensus.cc:2454] T 6a2b6a5f6d2e42c3be7e0bf4c4ce5f3f P 9e86ec40a36f496fbb75462f500f4035 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41789aa446ae4c5bbcda631441a01566 in term 1.
I20260620 09:26:11.945007 18726 raft_consensus.cc:2454] T 6a2b6a5f6d2e42c3be7e0bf4c4ce5f3f P b9807e3fbedb4973a065b0c81113fa6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41789aa446ae4c5bbcda631441a01566 in term 1.
I20260620 09:26:11.945137 18519 leader_election.cc:302] T 6a2b6a5f6d2e42c3be7e0bf4c4ce5f3f P 41789aa446ae4c5bbcda631441a01566 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41789aa446ae4c5bbcda631441a01566, b9807e3fbedb4973a065b0c81113fa6a; no voters:
I20260620 09:26:11.945267 18801 raft_consensus.cc:2781] T 6a2b6a5f6d2e42c3be7e0bf4c4ce5f3f P 41789aa446ae4c5bbcda631441a01566 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:26:11.945425 18801 raft_consensus.cc:686] T 6a2b6a5f6d2e42c3be7e0bf4c4ce5f3f P 41789aa446ae4c5bbcda631441a01566 [term 1 LEADER]: Becoming Leader. State: Replica: 41789aa446ae4c5bbcda631441a01566, State: Running, Role: LEADER
I20260620 09:26:11.945528 18801 consensus_queue.cc:237] T 6a2b6a5f6d2e42c3be7e0bf4c4ce5f3f P 41789aa446ae4c5bbcda631441a01566 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e86ec40a36f496fbb75462f500f4035" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 42805 } } peers { permanent_uuid: "41789aa446ae4c5bbcda631441a01566" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 41977 } } peers { permanent_uuid: "b9807e3fbedb4973a065b0c81113fa6a" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 41849 } }
I20260620 09:26:11.946523 18288 catalog_manager.cc:5495] T 6a2b6a5f6d2e42c3be7e0bf4c4ce5f3f P 41789aa446ae4c5bbcda631441a01566 reported cstate change: term changed from 0 to 1, leader changed from <none> to 41789aa446ae4c5bbcda631441a01566 (127.16.43.130). New cstate: current_term: 1 leader_uuid: "41789aa446ae4c5bbcda631441a01566" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e86ec40a36f496fbb75462f500f4035" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 42805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "41789aa446ae4c5bbcda631441a01566" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 41977 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9807e3fbedb4973a065b0c81113fa6a" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 41849 } health_report { overall_health: UNKNOWN } } }
Jun 20 09:26:11 dist-test-slave-706q krb5kdc[18244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947571, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.190@KRBTEST.COM for kudu/127.16.43.190@KRBTEST.COM
I20260620 09:26:11.983489 18812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:11.976389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:42057 (local address 127.0.0.1:56556)
0620 09:26:11.976598 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:11.976609 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:11.976679 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:11.976808 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:11.976809 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:11.977195 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:11.977243 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:11.977246 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:11.977892 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:11.977893 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:11.978460 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:11.978462 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:11.978775 (+ 313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:11.978776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:11.978815 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:11.979452 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:26:11.979456 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:26:11.981008 (+ 1552us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:26:11.982437 (+ 1429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:11.982443 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:11.982446 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:11.982650 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:11.982828 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:11.982829 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:11.982829 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:11.982884 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:11.983128 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:26:11.983136 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:26:11.983192 (+ 56us) client_negotiation.cc:769] Sending connection context
0620 09:26:11.983268 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 09:26:11.983373 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":93,"threads_started":1}
I20260620 09:26:11.983503 18356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:11.976516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56556 (local address 127.16.43.190:42057)
0620 09:26:11.976614 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:11.976616 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:11.976679 (+ 63us) server_negotiation.cc:407] Connection header received
0620 09:26:11.976725 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:11.976726 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:11.976736 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:11.976789 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:11.977331 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:11.977826 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:11.978536 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:11.978712 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:11.978762 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:11.981092 (+ 2330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:11.981093 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:11.981093 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:26:11.981099 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:11.982347 (+ 1248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:11.982720 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:11.982721 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:11.982721 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:11.982770 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:11.982936 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:11.982936 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:11.982936 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:11.983036 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:11.983099 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:11.983308 (+ 209us) server_negotiation.cc:299] Negotiation successful
0620 09:26:11.983421 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":43,"server-negotiator.queue_time_us":75}
I20260620 09:26:11.985891 18288 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.16.43.190@KRBTEST.COM'} at 127.0.0.1:56556:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 09:26:11.986183 18288 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 09:26:11.987958 18566 tablet_service.cc:1467] Processing CreateTablet for tablet 5710db528f5845d684e582fbca94f288 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3e9f005cc93744d7a1b8be0797da6969]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 09:26:11.987993 18426 tablet_service.cc:1467] Processing CreateTablet for tablet 5710db528f5845d684e582fbca94f288 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3e9f005cc93744d7a1b8be0797da6969]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 09:26:11.988051 18706 tablet_service.cc:1467] Processing CreateTablet for tablet 5710db528f5845d684e582fbca94f288 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3e9f005cc93744d7a1b8be0797da6969]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 09:26:11.988152 18566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5710db528f5845d684e582fbca94f288. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:26:11.988175 18706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5710db528f5845d684e582fbca94f288. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:26:11.988181 18426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5710db528f5845d684e582fbca94f288. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:26:11.989768 18795 tablet_bootstrap.cc:492] T 5710db528f5845d684e582fbca94f288 P b9807e3fbedb4973a065b0c81113fa6a: Bootstrap starting.
I20260620 09:26:11.989794 18799 tablet_bootstrap.cc:492] T 5710db528f5845d684e582fbca94f288 P 9e86ec40a36f496fbb75462f500f4035: Bootstrap starting.
I20260620 09:26:11.989933 18796 tablet_bootstrap.cc:492] T 5710db528f5845d684e582fbca94f288 P 41789aa446ae4c5bbcda631441a01566: Bootstrap starting.
I20260620 09:26:11.990368 18795 tablet_bootstrap.cc:654] T 5710db528f5845d684e582fbca94f288 P b9807e3fbedb4973a065b0c81113fa6a: Neither blocks nor log segments found. Creating new log.
I20260620 09:26:11.990401 18796 tablet_bootstrap.cc:654] T 5710db528f5845d684e582fbca94f288 P 41789aa446ae4c5bbcda631441a01566: Neither blocks nor log segments found. Creating new log.
I20260620 09:26:11.990516 18799 tablet_bootstrap.cc:654] T 5710db528f5845d684e582fbca94f288 P 9e86ec40a36f496fbb75462f500f4035: Neither blocks nor log segments found. Creating new log.
I20260620 09:26:11.991128 18795 tablet_bootstrap.cc:492] T 5710db528f5845d684e582fbca94f288 P b9807e3fbedb4973a065b0c81113fa6a: No bootstrap required, opened a new log
I20260620 09:26:11.991151 18796 tablet_bootstrap.cc:492] T 5710db528f5845d684e582fbca94f288 P 41789aa446ae4c5bbcda631441a01566: No bootstrap required, opened a new log
I20260620 09:26:11.991154 18799 tablet_bootstrap.cc:492] T 5710db528f5845d684e582fbca94f288 P 9e86ec40a36f496fbb75462f500f4035: No bootstrap required, opened a new log
I20260620 09:26:11.991201 18796 ts_tablet_manager.cc:1397] T 5710db528f5845d684e582fbca94f288 P 41789aa446ae4c5bbcda631441a01566: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260620 09:26:11.991201 18799 ts_tablet_manager.cc:1397] T 5710db528f5845d684e582fbca94f288 P 9e86ec40a36f496fbb75462f500f4035: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 09:26:11.991204 18795 ts_tablet_manager.cc:1397] T 5710db528f5845d684e582fbca94f288 P b9807e3fbedb4973a065b0c81113fa6a: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260620 09:26:11.991353 18799 raft_consensus.cc:348] T 5710db528f5845d684e582fbca94f288 P 9e86ec40a36f496fbb75462f500f4035 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e86ec40a36f496fbb75462f500f4035" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 42805 } } peers { permanent_uuid: "41789aa446ae4c5bbcda631441a01566" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 41977 } } peers { permanent_uuid: "b9807e3fbedb4973a065b0c81113fa6a" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 41849 } }
I20260620 09:26:11.991353 18796 raft_consensus.cc:348] T 5710db528f5845d684e582fbca94f288 P 41789aa446ae4c5bbcda631441a01566 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e86ec40a36f496fbb75462f500f4035" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 42805 } } peers { permanent_uuid: "41789aa446ae4c5bbcda631441a01566" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 41977 } } peers { permanent_uuid: "b9807e3fbedb4973a065b0c81113fa6a" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 41849 } }
I20260620 09:26:11.991436 18799 raft_consensus.cc:374] T 5710db528f5845d684e582fbca94f288 P 9e86ec40a36f496fbb75462f500f4035 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:26:11.991436 18796 raft_consensus.cc:374] T 5710db528f5845d684e582fbca94f288 P 41789aa446ae4c5bbcda631441a01566 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:26:11.991459 18799 raft_consensus.cc:729] T 5710db528f5845d684e582fbca94f288 P 9e86ec40a36f496fbb75462f500f4035 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e86ec40a36f496fbb75462f500f4035, State: Initialized, Role: FOLLOWER
I20260620 09:26:11.991387 18795 raft_consensus.cc:348] T 5710db528f5845d684e582fbca94f288 P b9807e3fbedb4973a065b0c81113fa6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e86ec40a36f496fbb75462f500f4035" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 42805 } } peers { permanent_uuid: "41789aa446ae4c5bbcda631441a01566" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 41977 } } peers { permanent_uuid: "b9807e3fbedb4973a065b0c81113fa6a" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 41849 } }
I20260620 09:26:11.991474 18796 raft_consensus.cc:729] T 5710db528f5845d684e582fbca94f288 P 41789aa446ae4c5bbcda631441a01566 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41789aa446ae4c5bbcda631441a01566, State: Initialized, Role: FOLLOWER
I20260620 09:26:11.991493 18795 raft_consensus.cc:374] T 5710db528f5845d684e582fbca94f288 P b9807e3fbedb4973a065b0c81113fa6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:26:11.991526 18795 raft_consensus.cc:729] T 5710db528f5845d684e582fbca94f288 P b9807e3fbedb4973a065b0c81113fa6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9807e3fbedb4973a065b0c81113fa6a, State: Initialized, Role: FOLLOWER
I20260620 09:26:11.991503 18799 consensus_queue.cc:260] T 5710db528f5845d684e582fbca94f288 P 9e86ec40a36f496fbb75462f500f4035 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e86ec40a36f496fbb75462f500f4035" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 42805 } } peers { permanent_uuid: "41789aa446ae4c5bbcda631441a01566" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 41977 } } peers { permanent_uuid: "b9807e3fbedb4973a065b0c81113fa6a" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 41849 } }
I20260620 09:26:11.991542 18796 consensus_queue.cc:260] T 5710db528f5845d684e582fbca94f288 P 41789aa446ae4c5bbcda631441a01566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e86ec40a36f496fbb75462f500f4035" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 42805 } } peers { permanent_uuid: "41789aa446ae4c5bbcda631441a01566" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 41977 } } peers { permanent_uuid: "b9807e3fbedb4973a065b0c81113fa6a" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 41849 } }
I20260620 09:26:11.991595 18795 consensus_queue.cc:260] T 5710db528f5845d684e582fbca94f288 P b9807e3fbedb4973a065b0c81113fa6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e86ec40a36f496fbb75462f500f4035" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 42805 } } peers { permanent_uuid: "41789aa446ae4c5bbcda631441a01566" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 41977 } } peers { permanent_uuid: "b9807e3fbedb4973a065b0c81113fa6a" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 41849 } }
I20260620 09:26:11.991626 18799 ts_tablet_manager.cc:1428] T 5710db528f5845d684e582fbca94f288 P 9e86ec40a36f496fbb75462f500f4035: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 09:26:11.991639 18796 ts_tablet_manager.cc:1428] T 5710db528f5845d684e582fbca94f288 P 41789aa446ae4c5bbcda631441a01566: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 09:26:11.991700 18795 ts_tablet_manager.cc:1428] T 5710db528f5845d684e582fbca94f288 P b9807e3fbedb4973a065b0c81113fa6a: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 09:26:11.991757 18801 tablet_replica.cc:439] T 5710db528f5845d684e582fbca94f288 P 41789aa446ae4c5bbcda631441a01566: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e86ec40a36f496fbb75462f500f4035" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 42805 } } peers { permanent_uuid: "41789aa446ae4c5bbcda631441a01566" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 41977 } } peers { permanent_uuid: "b9807e3fbedb4973a065b0c81113fa6a" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 41849 } } }
I20260620 09:26:11.991829 18801 tablet_replica.cc:442] T 5710db528f5845d684e582fbca94f288 P 41789aa446ae4c5bbcda631441a01566: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 09:26:11.991863 18803 tablet_replica.cc:439] T 5710db528f5845d684e582fbca94f288 P 9e86ec40a36f496fbb75462f500f4035: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e86ec40a36f496fbb75462f500f4035" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 42805 } } peers { permanent_uuid: "41789aa446ae4c5bbcda631441a01566" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 41977 } } peers { permanent_uuid: "b9807e3fbedb4973a065b0c81113fa6a" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 41849 } } }
I20260620 09:26:11.991959 18803 tablet_replica.cc:442] T 5710db528f5845d684e582fbca94f288 P 9e86ec40a36f496fbb75462f500f4035: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 09:26:11.992127 18800 tablet_replica.cc:439] T 5710db528f5845d684e582fbca94f288 P b9807e3fbedb4973a065b0c81113fa6a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e86ec40a36f496fbb75462f500f4035" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 42805 } } peers { permanent_uuid: "41789aa446ae4c5bbcda631441a01566" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 41977 } } peers { permanent_uuid: "b9807e3fbedb4973a065b0c81113fa6a" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 41849 } } }
I20260620 09:26:11.992228 18800 tablet_replica.cc:442] T 5710db528f5845d684e582fbca94f288 P b9807e3fbedb4973a065b0c81113fa6a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 09:26:11.994946 18801 raft_consensus.cc:482] T 5710db528f5845d684e582fbca94f288 P 41789aa446ae4c5bbcda631441a01566 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:26:11.995008 18801 raft_consensus.cc:504] T 5710db528f5845d684e582fbca94f288 P 41789aa446ae4c5bbcda631441a01566 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e86ec40a36f496fbb75462f500f4035" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 42805 } } peers { permanent_uuid: "41789aa446ae4c5bbcda631441a01566" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 41977 } } peers { permanent_uuid: "b9807e3fbedb4973a065b0c81113fa6a" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 41849 } }
I20260620 09:26:11.995172 18801 leader_election.cc:288] T 5710db528f5845d684e582fbca94f288 P 41789aa446ae4c5bbcda631441a01566 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9e86ec40a36f496fbb75462f500f4035 (127.16.43.129:42805), b9807e3fbedb4973a065b0c81113fa6a (127.16.43.131:41849)
I20260620 09:26:11.995329 18726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5710db528f5845d684e582fbca94f288" candidate_uuid: "41789aa446ae4c5bbcda631441a01566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9807e3fbedb4973a065b0c81113fa6a" is_pre_election: true
I20260620 09:26:11.995414 18446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5710db528f5845d684e582fbca94f288" candidate_uuid: "41789aa446ae4c5bbcda631441a01566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e86ec40a36f496fbb75462f500f4035" is_pre_election: true
I20260620 09:26:11.995455 18726 raft_consensus.cc:2454] T 5710db528f5845d684e582fbca94f288 P b9807e3fbedb4973a065b0c81113fa6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41789aa446ae4c5bbcda631441a01566 in term 0.
I20260620 09:26:11.995486 18446 raft_consensus.cc:2454] T 5710db528f5845d684e582fbca94f288 P 9e86ec40a36f496fbb75462f500f4035 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41789aa446ae4c5bbcda631441a01566 in term 0.
I20260620 09:26:11.995630 18519 leader_election.cc:302] T 5710db528f5845d684e582fbca94f288 P 41789aa446ae4c5bbcda631441a01566 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41789aa446ae4c5bbcda631441a01566, b9807e3fbedb4973a065b0c81113fa6a; no voters:
I20260620 09:26:11.995718 18801 raft_consensus.cc:2781] T 5710db528f5845d684e582fbca94f288 P 41789aa446ae4c5bbcda631441a01566 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 09:26:11.995759 18801 raft_consensus.cc:482] T 5710db528f5845d684e582fbca94f288 P 41789aa446ae4c5bbcda631441a01566 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 09:26:11.995790 18801 raft_consensus.cc:3037] T 5710db528f5845d684e582fbca94f288 P 41789aa446ae4c5bbcda631441a01566 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:26:11.996372 18801 raft_consensus.cc:504] T 5710db528f5845d684e582fbca94f288 P 41789aa446ae4c5bbcda631441a01566 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e86ec40a36f496fbb75462f500f4035" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 42805 } } peers { permanent_uuid: "41789aa446ae4c5bbcda631441a01566" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 41977 } } peers { permanent_uuid: "b9807e3fbedb4973a065b0c81113fa6a" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 41849 } }
I20260620 09:26:11.996518 18801 leader_election.cc:288] T 5710db528f5845d684e582fbca94f288 P 41789aa446ae4c5bbcda631441a01566 [CANDIDATE]: Term 1 election: Requested vote from peers 9e86ec40a36f496fbb75462f500f4035 (127.16.43.129:42805), b9807e3fbedb4973a065b0c81113fa6a (127.16.43.131:41849)
I20260620 09:26:11.996656 18446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5710db528f5845d684e582fbca94f288" candidate_uuid: "41789aa446ae4c5bbcda631441a01566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e86ec40a36f496fbb75462f500f4035"
I20260620 09:26:11.996745 18446 raft_consensus.cc:3037] T 5710db528f5845d684e582fbca94f288 P 9e86ec40a36f496fbb75462f500f4035 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:26:11.997395 18446 raft_consensus.cc:2454] T 5710db528f5845d684e582fbca94f288 P 9e86ec40a36f496fbb75462f500f4035 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41789aa446ae4c5bbcda631441a01566 in term 1.
I20260620 09:26:11.997599 18518 leader_election.cc:302] T 5710db528f5845d684e582fbca94f288 P 41789aa446ae4c5bbcda631441a01566 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41789aa446ae4c5bbcda631441a01566, 9e86ec40a36f496fbb75462f500f4035; no voters:
I20260620 09:26:11.997628 18726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5710db528f5845d684e582fbca94f288" candidate_uuid: "41789aa446ae4c5bbcda631441a01566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9807e3fbedb4973a065b0c81113fa6a"
I20260620 09:26:11.997711 18726 raft_consensus.cc:3037] T 5710db528f5845d684e582fbca94f288 P b9807e3fbedb4973a065b0c81113fa6a [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:26:11.997712 18801 raft_consensus.cc:2781] T 5710db528f5845d684e582fbca94f288 P 41789aa446ae4c5bbcda631441a01566 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:26:11.997754 18801 raft_consensus.cc:686] T 5710db528f5845d684e582fbca94f288 P 41789aa446ae4c5bbcda631441a01566 [term 1 LEADER]: Becoming Leader. State: Replica: 41789aa446ae4c5bbcda631441a01566, State: Running, Role: LEADER
I20260620 09:26:11.997805 18801 consensus_queue.cc:237] T 5710db528f5845d684e582fbca94f288 P 41789aa446ae4c5bbcda631441a01566 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e86ec40a36f496fbb75462f500f4035" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 42805 } } peers { permanent_uuid: "41789aa446ae4c5bbcda631441a01566" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 41977 } } peers { permanent_uuid: "b9807e3fbedb4973a065b0c81113fa6a" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 41849 } }
I20260620 09:26:11.998116 18801 tablet_replica.cc:439] T 5710db528f5845d684e582fbca94f288 P 41789aa446ae4c5bbcda631441a01566: TxnStatusTablet state changed. Reason: New leader 41789aa446ae4c5bbcda631441a01566. Latest consensus state: current_term: 1 leader_uuid: "41789aa446ae4c5bbcda631441a01566" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e86ec40a36f496fbb75462f500f4035" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 42805 } } peers { permanent_uuid: "41789aa446ae4c5bbcda631441a01566" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 41977 } } peers { permanent_uuid: "b9807e3fbedb4973a065b0c81113fa6a" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 41849 } } }
I20260620 09:26:11.998193 18801 tablet_replica.cc:442] T 5710db528f5845d684e582fbca94f288 P 41789aa446ae4c5bbcda631441a01566: This TxnStatusTablet replica's current role is: LEADER
I20260620 09:26:11.998415 18817 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 09:26:11.998530 18726 raft_consensus.cc:2454] T 5710db528f5845d684e582fbca94f288 P b9807e3fbedb4973a065b0c81113fa6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41789aa446ae4c5bbcda631441a01566 in term 1.
I20260620 09:26:11.998675 18817 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 09:26:12.000970 18287 catalog_manager.cc:5495] T 5710db528f5845d684e582fbca94f288 P 41789aa446ae4c5bbcda631441a01566 reported cstate change: term changed from 0 to 1, leader changed from <none> to 41789aa446ae4c5bbcda631441a01566 (127.16.43.130). New cstate: current_term: 1 leader_uuid: "41789aa446ae4c5bbcda631441a01566" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e86ec40a36f496fbb75462f500f4035" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 42805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "41789aa446ae4c5bbcda631441a01566" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 41977 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9807e3fbedb4973a065b0c81113fa6a" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 41849 } health_report { overall_health: UNKNOWN } } }
I20260620 09:26:12.008311 18792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:12.005464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53956 (local address 127.16.43.130:41977)
0620 09:26:12.005525 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:12.005527 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:12.005536 (+ 9us) server_negotiation.cc:407] Connection header received
0620 09:26:12.005621 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:12.005621 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:12.005636 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:12.005693 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:26:12.005817 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:12.006310 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:12.007450 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:12.007721 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:12.007775 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:12.007992 (+ 217us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:26:12.008086 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:26:12.008146 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:12.008232 (+ 86us) server_negotiation.cc:299] Negotiation successful
0620 09:26:12.008251 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 09:26:12.008373 18812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:12.005397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.130:41977 (local address 127.0.0.1:53956)
0620 09:26:12.005479 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:12.005492 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:12.005538 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:12.005699 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:12.005700 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:12.005701 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 09:26:12.005754 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:12.005756 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:12.006402 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:12.006404 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:12.007357 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:12.007360 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:12.007797 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:12.007798 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:12.007871 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:12.007892 (+ 21us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 09:26:12.008152 (+ 260us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 09:26:12.008161 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 09:26:12.008230 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 09:26:12.008282 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260620 09:26:12.010223 18726 raft_consensus.cc:1261] T 5710db528f5845d684e582fbca94f288 P b9807e3fbedb4973a065b0c81113fa6a [term 1 FOLLOWER]: Refusing update from remote peer 41789aa446ae4c5bbcda631441a01566: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:26:12.010244 18446 raft_consensus.cc:1261] T 5710db528f5845d684e582fbca94f288 P 9e86ec40a36f496fbb75462f500f4035 [term 1 FOLLOWER]: Refusing update from remote peer 41789aa446ae4c5bbcda631441a01566: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:26:12.010586 18801 consensus_queue.cc:1035] T 5710db528f5845d684e582fbca94f288 P 41789aa446ae4c5bbcda631441a01566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e86ec40a36f496fbb75462f500f4035" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 42805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:26:12.010735 18805 consensus_queue.cc:1035] T 5710db528f5845d684e582fbca94f288 P 41789aa446ae4c5bbcda631441a01566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9807e3fbedb4973a065b0c81113fa6a" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 41849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:26:12.012257 18803 tablet_replica.cc:439] T 5710db528f5845d684e582fbca94f288 P 9e86ec40a36f496fbb75462f500f4035: TxnStatusTablet state changed. Reason: New leader 41789aa446ae4c5bbcda631441a01566. Latest consensus state: current_term: 1 leader_uuid: "41789aa446ae4c5bbcda631441a01566" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e86ec40a36f496fbb75462f500f4035" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 42805 } } peers { permanent_uuid: "41789aa446ae4c5bbcda631441a01566" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 41977 } } peers { permanent_uuid: "b9807e3fbedb4973a065b0c81113fa6a" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 41849 } } }
I20260620 09:26:12.012403 18803 tablet_replica.cc:442] T 5710db528f5845d684e582fbca94f288 P 9e86ec40a36f496fbb75462f500f4035: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 09:26:12.012385 18800 tablet_replica.cc:439] T 5710db528f5845d684e582fbca94f288 P b9807e3fbedb4973a065b0c81113fa6a: TxnStatusTablet state changed. Reason: New leader 41789aa446ae4c5bbcda631441a01566. Latest consensus state: current_term: 1 leader_uuid: "41789aa446ae4c5bbcda631441a01566" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e86ec40a36f496fbb75462f500f4035" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 42805 } } peers { permanent_uuid: "41789aa446ae4c5bbcda631441a01566" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 41977 } } peers { permanent_uuid: "b9807e3fbedb4973a065b0c81113fa6a" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 41849 } } }
I20260620 09:26:12.012488 18800 tablet_replica.cc:442] T 5710db528f5845d684e582fbca94f288 P b9807e3fbedb4973a065b0c81113fa6a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 09:26:12.013551 18801 tablet_replica.cc:439] T 5710db528f5845d684e582fbca94f288 P 41789aa446ae4c5bbcda631441a01566: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "41789aa446ae4c5bbcda631441a01566" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e86ec40a36f496fbb75462f500f4035" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 42805 } } peers { permanent_uuid: "41789aa446ae4c5bbcda631441a01566" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 41977 } } peers { permanent_uuid: "b9807e3fbedb4973a065b0c81113fa6a" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 41849 } } }
I20260620 09:26:12.013650 18801 tablet_replica.cc:442] T 5710db528f5845d684e582fbca94f288 P 41789aa446ae4c5bbcda631441a01566: This TxnStatusTablet replica's current role is: LEADER
I20260620 09:26:12.014187 18803 tablet_replica.cc:439] T 5710db528f5845d684e582fbca94f288 P 9e86ec40a36f496fbb75462f500f4035: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "41789aa446ae4c5bbcda631441a01566" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e86ec40a36f496fbb75462f500f4035" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 42805 } } peers { permanent_uuid: "41789aa446ae4c5bbcda631441a01566" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 41977 } } peers { permanent_uuid: "b9807e3fbedb4973a065b0c81113fa6a" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 41849 } } }
I20260620 09:26:12.014272 18803 tablet_replica.cc:442] T 5710db528f5845d684e582fbca94f288 P 9e86ec40a36f496fbb75462f500f4035: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 09:26:12.014861 18805 tablet_replica.cc:439] T 5710db528f5845d684e582fbca94f288 P 41789aa446ae4c5bbcda631441a01566: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "41789aa446ae4c5bbcda631441a01566" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e86ec40a36f496fbb75462f500f4035" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 42805 } } peers { permanent_uuid: "41789aa446ae4c5bbcda631441a01566" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 41977 } } peers { permanent_uuid: "b9807e3fbedb4973a065b0c81113fa6a" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 41849 } } }
I20260620 09:26:12.014966 18805 tablet_replica.cc:442] T 5710db528f5845d684e582fbca94f288 P 41789aa446ae4c5bbcda631441a01566: This TxnStatusTablet replica's current role is: LEADER
I20260620 09:26:12.015580 18800 tablet_replica.cc:439] T 5710db528f5845d684e582fbca94f288 P b9807e3fbedb4973a065b0c81113fa6a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "41789aa446ae4c5bbcda631441a01566" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e86ec40a36f496fbb75462f500f4035" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 42805 } } peers { permanent_uuid: "41789aa446ae4c5bbcda631441a01566" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 41977 } } peers { permanent_uuid: "b9807e3fbedb4973a065b0c81113fa6a" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 41849 } } }
I20260620 09:26:12.015668 18800 tablet_replica.cc:442] T 5710db528f5845d684e582fbca94f288 P b9807e3fbedb4973a065b0c81113fa6a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 09:26:12.015646 18821 mvcc.cc:205] Tried to move back new op lower bound from 7298857254950744064 to 7298857254903341056. Current Snapshot: MvccSnapshot[applied={T|T < 7298857254950744064 or (T in {7298857254950744064})}]
I20260620 09:26:12.015882 18820 mvcc.cc:205] Tried to move back new op lower bound from 7298857254950744064 to 7298857254903341056. Current Snapshot: MvccSnapshot[applied={T|T < 7298857254950744064 or (T in {7298857254950744064})}]
I20260620 09:26:12.016064 18822 mvcc.cc:205] Tried to move back new op lower bound from 7298857254950744064 to 7298857254903341056. Current Snapshot: MvccSnapshot[applied={T|T < 7298857254950744064}]
I20260620 09:26:12.019101 18792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:12.016490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53960 (local address 127.16.43.130:41977)
0620 09:26:12.016526 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:12.016528 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:12.016535 (+ 7us) server_negotiation.cc:407] Connection header received
0620 09:26:12.016566 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:12.016567 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:12.016578 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:12.016627 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:26:12.016792 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:12.017546 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:12.018382 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:12.018685 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:12.018766 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:12.018865 (+ 99us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:26:12.018900 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:26:12.018956 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:12.019005 (+ 49us) server_negotiation.cc:299] Negotiation successful
0620 09:26:12.019026 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
W20260620 09:26:12.020870 18493 tablet.cc:2369] T 6a2b6a5f6d2e42c3be7e0bf4c4ce5f3f P 9e86ec40a36f496fbb75462f500f4035: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 09:26:12.024713 18638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:12.021984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.130:41977 (local address 127.16.43.130:33685)
0620 09:26:12.022072 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:12.022086 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:12.022134 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:12.022294 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:12.022295 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:12.022297 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 09:26:12.022347 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:12.022349 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:12.023225 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:12.023226 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:12.023875 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:12.023878 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:12.024266 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:12.024267 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:12.024327 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:12.024341 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 09:26:12.024534 (+ 193us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 09:26:12.024536 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 09:26:12.024611 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 09:26:12.024653 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47,"mutex_wait_us":2}
I20260620 09:26:12.024714 18792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:12.022014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.130:33685 (local address 127.16.43.130:41977)
0620 09:26:12.022200 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:12.022202 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:12.022207 (+ 5us) server_negotiation.cc:407] Connection header received
0620 09:26:12.022223 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:12.022223 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:12.022231 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:12.022280 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:26:12.022423 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:12.023145 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:12.023982 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:12.024167 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:12.024254 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:12.024437 (+ 183us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:26:12.024475 (+ 38us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:26:12.024535 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:12.024611 (+ 76us) server_negotiation.cc:299] Negotiation successful
0620 09:26:12.024633 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172}
I20260620 09:26:12.026949 18446 raft_consensus.cc:1261] T 6a2b6a5f6d2e42c3be7e0bf4c4ce5f3f P 9e86ec40a36f496fbb75462f500f4035 [term 1 FOLLOWER]: Refusing update from remote peer 41789aa446ae4c5bbcda631441a01566: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:26:12.026985 18726 raft_consensus.cc:1261] T 6a2b6a5f6d2e42c3be7e0bf4c4ce5f3f P b9807e3fbedb4973a065b0c81113fa6a [term 1 FOLLOWER]: Refusing update from remote peer 41789aa446ae4c5bbcda631441a01566: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:26:12.027143 18805 consensus_queue.cc:1035] T 6a2b6a5f6d2e42c3be7e0bf4c4ce5f3f P 41789aa446ae4c5bbcda631441a01566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e86ec40a36f496fbb75462f500f4035" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 42805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:26:12.027240 18827 consensus_queue.cc:1035] T 6a2b6a5f6d2e42c3be7e0bf4c4ce5f3f P 41789aa446ae4c5bbcda631441a01566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9807e3fbedb4973a065b0c81113fa6a" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 41849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:26:12.041713 18287 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56554:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:26:12.041843 18287 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56554:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:26:12.042629 18287 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 21cc679fd523463da96d6adcab8040db: Sending DeleteTablet for 3 replicas of tablet 6a2b6a5f6d2e42c3be7e0bf4c4ce5f3f
I20260620 09:26:12.042955 18426 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a2b6a5f6d2e42c3be7e0bf4c4ce5f3f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:26:12 UTC) from {username='kudu', principal='kudu/127.16.43.190@KRBTEST.COM'} at 127.0.0.1:53164
I20260620 09:26:12.042966 18565 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a2b6a5f6d2e42c3be7e0bf4c4ce5f3f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:26:12 UTC) from {username='kudu', principal='kudu/127.16.43.190@KRBTEST.COM'} at 127.0.0.1:53952
I20260620 09:26:12.042985 18706 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a2b6a5f6d2e42c3be7e0bf4c4ce5f3f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:26:12 UTC) from {username='kudu', principal='kudu/127.16.43.190@KRBTEST.COM'} at 127.0.0.1:52330
I20260620 09:26:12.043216 18837 tablet_replica.cc:330] T 6a2b6a5f6d2e42c3be7e0bf4c4ce5f3f P b9807e3fbedb4973a065b0c81113fa6a: stopping tablet replica
I20260620 09:26:12.043319 18837 raft_consensus.cc:2229] T 6a2b6a5f6d2e42c3be7e0bf4c4ce5f3f P b9807e3fbedb4973a065b0c81113fa6a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:26:12.043388 18837 raft_consensus.cc:2258] T 6a2b6a5f6d2e42c3be7e0bf4c4ce5f3f P b9807e3fbedb4973a065b0c81113fa6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:26:12.043416 18287 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56554:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 09:26:12.043596 18287 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 09:26:12.043855 18837 ts_tablet_manager.cc:1905] T 6a2b6a5f6d2e42c3be7e0bf4c4ce5f3f P b9807e3fbedb4973a065b0c81113fa6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:26:12.043898 18835 tablet_replica.cc:330] T 6a2b6a5f6d2e42c3be7e0bf4c4ce5f3f P 41789aa446ae4c5bbcda631441a01566: stopping tablet replica
I20260620 09:26:12.044011 18835 raft_consensus.cc:2229] T 6a2b6a5f6d2e42c3be7e0bf4c4ce5f3f P 41789aa446ae4c5bbcda631441a01566 [term 1 LEADER]: Raft consensus shutting down.
I20260620 09:26:12.044790 18426 tablet_service.cc:1467] Processing CreateTablet for tablet f3f67960ea2f4393902c37160f338873 (DEFAULT_TABLE table=test-table [id=19c71355bd0b4886a4580159543ba8bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:26:12.044898 18565 tablet_service.cc:1467] Processing CreateTablet for tablet f3f67960ea2f4393902c37160f338873 (DEFAULT_TABLE table=test-table [id=19c71355bd0b4886a4580159543ba8bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:26:12.044921 18426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3f67960ea2f4393902c37160f338873. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:26:12.044988 18565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3f67960ea2f4393902c37160f338873. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:26:12.045759 18837 ts_tablet_manager.cc:1918] T 6a2b6a5f6d2e42c3be7e0bf4c4ce5f3f P b9807e3fbedb4973a065b0c81113fa6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 09:26:12.045825 18837 log.cc:1196] T 6a2b6a5f6d2e42c3be7e0bf4c4ce5f3f P b9807e3fbedb4973a065b0c81113fa6a: Deleting WAL directory at /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781947562949448-16558-0/minicluster-data/ts-2/wal/wals/6a2b6a5f6d2e42c3be7e0bf4c4ce5f3f
I20260620 09:26:12.046085 18837 ts_tablet_manager.cc:1939] T 6a2b6a5f6d2e42c3be7e0bf4c4ce5f3f P b9807e3fbedb4973a065b0c81113fa6a: Deleting consensus metadata
I20260620 09:26:12.046397 18799 tablet_bootstrap.cc:492] T f3f67960ea2f4393902c37160f338873 P 9e86ec40a36f496fbb75462f500f4035: Bootstrap starting.
I20260620 09:26:12.046523 18706 tablet_service.cc:1467] Processing CreateTablet for tablet f3f67960ea2f4393902c37160f338873 (DEFAULT_TABLE table=test-table [id=19c71355bd0b4886a4580159543ba8bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:26:12.046640 18706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3f67960ea2f4393902c37160f338873. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:26:12.046789 18799 tablet_bootstrap.cc:654] T f3f67960ea2f4393902c37160f338873 P 9e86ec40a36f496fbb75462f500f4035: Neither blocks nor log segments found. Creating new log.
I20260620 09:26:12.047407 18799 tablet_bootstrap.cc:492] T f3f67960ea2f4393902c37160f338873 P 9e86ec40a36f496fbb75462f500f4035: No bootstrap required, opened a new log
I20260620 09:26:12.047456 18799 ts_tablet_manager.cc:1397] T f3f67960ea2f4393902c37160f338873 P 9e86ec40a36f496fbb75462f500f4035: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 09:26:12.047633 18799 raft_consensus.cc:348] T f3f67960ea2f4393902c37160f338873 P 9e86ec40a36f496fbb75462f500f4035 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e86ec40a36f496fbb75462f500f4035" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 42805 } } peers { permanent_uuid: "41789aa446ae4c5bbcda631441a01566" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 41977 } } peers { permanent_uuid: "b9807e3fbedb4973a065b0c81113fa6a" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 41849 } }
I20260620 09:26:12.047708 18799 raft_consensus.cc:374] T f3f67960ea2f4393902c37160f338873 P 9e86ec40a36f496fbb75462f500f4035 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:26:12.047736 18799 raft_consensus.cc:729] T f3f67960ea2f4393902c37160f338873 P 9e86ec40a36f496fbb75462f500f4035 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e86ec40a36f496fbb75462f500f4035, State: Initialized, Role: FOLLOWER
I20260620 09:26:12.047778 18799 consensus_queue.cc:260] T f3f67960ea2f4393902c37160f338873 P 9e86ec40a36f496fbb75462f500f4035 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e86ec40a36f496fbb75462f500f4035" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 42805 } } peers { permanent_uuid: "41789aa446ae4c5bbcda631441a01566" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 41977 } } peers { permanent_uuid: "b9807e3fbedb4973a065b0c81113fa6a" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 41849 } }
I20260620 09:26:12.047864 18799 ts_tablet_manager.cc:1428] T f3f67960ea2f4393902c37160f338873 P 9e86ec40a36f496fbb75462f500f4035: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260620 09:26:12.047923 18795 tablet_bootstrap.cc:492] T f3f67960ea2f4393902c37160f338873 P b9807e3fbedb4973a065b0c81113fa6a: Bootstrap starting.
I20260620 09:26:12.047952 18796 tablet_bootstrap.cc:492] T f3f67960ea2f4393902c37160f338873 P 41789aa446ae4c5bbcda631441a01566: Bootstrap starting.
I20260620 09:26:12.048334 18796 tablet_bootstrap.cc:654] T f3f67960ea2f4393902c37160f338873 P 41789aa446ae4c5bbcda631441a01566: Neither blocks nor log segments found. Creating new log.
I20260620 09:26:12.048334 18795 tablet_bootstrap.cc:654] T f3f67960ea2f4393902c37160f338873 P b9807e3fbedb4973a065b0c81113fa6a: Neither blocks nor log segments found. Creating new log.
I20260620 09:26:12.048813 18275 catalog_manager.cc:4841] TS b9807e3fbedb4973a065b0c81113fa6a (127.16.43.131:41849): tablet 6a2b6a5f6d2e42c3be7e0bf4c4ce5f3f (table test-table [id=3517409efffb41508f4b8e4245eb00bc]) successfully deleted
I20260620 09:26:12.049006 18795 tablet_bootstrap.cc:492] T f3f67960ea2f4393902c37160f338873 P b9807e3fbedb4973a065b0c81113fa6a: No bootstrap required, opened a new log
I20260620 09:26:12.049022 18835 raft_consensus.cc:2258] T 6a2b6a5f6d2e42c3be7e0bf4c4ce5f3f P 41789aa446ae4c5bbcda631441a01566 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:26:12.049077 18795 ts_tablet_manager.cc:1397] T f3f67960ea2f4393902c37160f338873 P b9807e3fbedb4973a065b0c81113fa6a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 09:26:12.049213 18795 raft_consensus.cc:348] T f3f67960ea2f4393902c37160f338873 P b9807e3fbedb4973a065b0c81113fa6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e86ec40a36f496fbb75462f500f4035" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 42805 } } peers { permanent_uuid: "41789aa446ae4c5bbcda631441a01566" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 41977 } } peers { permanent_uuid: "b9807e3fbedb4973a065b0c81113fa6a" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 41849 } }
I20260620 09:26:12.049280 18795 raft_consensus.cc:374] T f3f67960ea2f4393902c37160f338873 P b9807e3fbedb4973a065b0c81113fa6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:26:12.049305 18795 raft_consensus.cc:729] T f3f67960ea2f4393902c37160f338873 P b9807e3fbedb4973a065b0c81113fa6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9807e3fbedb4973a065b0c81113fa6a, State: Initialized, Role: FOLLOWER
I20260620 09:26:12.049355 18795 consensus_queue.cc:260] T f3f67960ea2f4393902c37160f338873 P b9807e3fbedb4973a065b0c81113fa6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e86ec40a36f496fbb75462f500f4035" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 42805 } } peers { permanent_uuid: "41789aa446ae4c5bbcda631441a01566" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 41977 } } peers { permanent_uuid: "b9807e3fbedb4973a065b0c81113fa6a" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 41849 } }
I20260620 09:26:12.049472 18795 ts_tablet_manager.cc:1428] T f3f67960ea2f4393902c37160f338873 P b9807e3fbedb4973a065b0c81113fa6a: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 09:26:12.049582 18835 ts_tablet_manager.cc:1905] T 6a2b6a5f6d2e42c3be7e0bf4c4ce5f3f P 41789aa446ae4c5bbcda631441a01566: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:26:12.050798 18836 tablet_replica.cc:330] T 6a2b6a5f6d2e42c3be7e0bf4c4ce5f3f P 9e86ec40a36f496fbb75462f500f4035: stopping tablet replica
I20260620 09:26:12.050957 18836 raft_consensus.cc:2229] T 6a2b6a5f6d2e42c3be7e0bf4c4ce5f3f P 9e86ec40a36f496fbb75462f500f4035 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:26:12.051038 18836 raft_consensus.cc:2258] T 6a2b6a5f6d2e42c3be7e0bf4c4ce5f3f P 9e86ec40a36f496fbb75462f500f4035 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:26:12.051195 18835 ts_tablet_manager.cc:1918] T 6a2b6a5f6d2e42c3be7e0bf4c4ce5f3f P 41789aa446ae4c5bbcda631441a01566: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 09:26:12.051263 18835 log.cc:1196] T 6a2b6a5f6d2e42c3be7e0bf4c4ce5f3f P 41789aa446ae4c5bbcda631441a01566: Deleting WAL directory at /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781947562949448-16558-0/minicluster-data/ts-1/wal/wals/6a2b6a5f6d2e42c3be7e0bf4c4ce5f3f
I20260620 09:26:12.051462 18836 ts_tablet_manager.cc:1905] T 6a2b6a5f6d2e42c3be7e0bf4c4ce5f3f P 9e86ec40a36f496fbb75462f500f4035: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:26:12.051568 18835 ts_tablet_manager.cc:1939] T 6a2b6a5f6d2e42c3be7e0bf4c4ce5f3f P 41789aa446ae4c5bbcda631441a01566: Deleting consensus metadata
I20260620 09:26:12.051934 18275 catalog_manager.cc:4841] TS 41789aa446ae4c5bbcda631441a01566 (127.16.43.130:41977): tablet 6a2b6a5f6d2e42c3be7e0bf4c4ce5f3f (table test-table [id=3517409efffb41508f4b8e4245eb00bc]) successfully deleted
I20260620 09:26:12.052955 18836 ts_tablet_manager.cc:1918] T 6a2b6a5f6d2e42c3be7e0bf4c4ce5f3f P 9e86ec40a36f496fbb75462f500f4035: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 09:26:12.053015 18836 log.cc:1196] T 6a2b6a5f6d2e42c3be7e0bf4c4ce5f3f P 9e86ec40a36f496fbb75462f500f4035: Deleting WAL directory at /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781947562949448-16558-0/minicluster-data/ts-0/wal/wals/6a2b6a5f6d2e42c3be7e0bf4c4ce5f3f
I20260620 09:26:12.053305 18836 ts_tablet_manager.cc:1939] T 6a2b6a5f6d2e42c3be7e0bf4c4ce5f3f P 9e86ec40a36f496fbb75462f500f4035: Deleting consensus metadata
I20260620 09:26:12.053704 18274 catalog_manager.cc:4841] TS 9e86ec40a36f496fbb75462f500f4035 (127.16.43.129:42805): tablet 6a2b6a5f6d2e42c3be7e0bf4c4ce5f3f (table test-table [id=3517409efffb41508f4b8e4245eb00bc]) successfully deleted
I20260620 09:26:12.055927 18796 tablet_bootstrap.cc:492] T f3f67960ea2f4393902c37160f338873 P 41789aa446ae4c5bbcda631441a01566: No bootstrap required, opened a new log
I20260620 09:26:12.055999 18796 ts_tablet_manager.cc:1397] T f3f67960ea2f4393902c37160f338873 P 41789aa446ae4c5bbcda631441a01566: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20260620 09:26:12.056144 18796 raft_consensus.cc:348] T f3f67960ea2f4393902c37160f338873 P 41789aa446ae4c5bbcda631441a01566 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e86ec40a36f496fbb75462f500f4035" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 42805 } } peers { permanent_uuid: "41789aa446ae4c5bbcda631441a01566" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 41977 } } peers { permanent_uuid: "b9807e3fbedb4973a065b0c81113fa6a" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 41849 } }
I20260620 09:26:12.056264 18796 raft_consensus.cc:374] T f3f67960ea2f4393902c37160f338873 P 41789aa446ae4c5bbcda631441a01566 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:26:12.056285 18796 raft_consensus.cc:729] T f3f67960ea2f4393902c37160f338873 P 41789aa446ae4c5bbcda631441a01566 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41789aa446ae4c5bbcda631441a01566, State: Initialized, Role: FOLLOWER
I20260620 09:26:12.056326 18796 consensus_queue.cc:260] T f3f67960ea2f4393902c37160f338873 P 41789aa446ae4c5bbcda631441a01566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e86ec40a36f496fbb75462f500f4035" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 42805 } } peers { permanent_uuid: "41789aa446ae4c5bbcda631441a01566" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 41977 } } peers { permanent_uuid: "b9807e3fbedb4973a065b0c81113fa6a" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 41849 } }
I20260620 09:26:12.056452 18796 ts_tablet_manager.cc:1428] T f3f67960ea2f4393902c37160f338873 P 41789aa446ae4c5bbcda631441a01566: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
W20260620 09:26:12.091630 18773 tablet.cc:2369] T f3f67960ea2f4393902c37160f338873 P b9807e3fbedb4973a065b0c81113fa6a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 09:26:12.149080 18800 raft_consensus.cc:482] T f3f67960ea2f4393902c37160f338873 P b9807e3fbedb4973a065b0c81113fa6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:26:12.149209 18800 raft_consensus.cc:504] T f3f67960ea2f4393902c37160f338873 P b9807e3fbedb4973a065b0c81113fa6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e86ec40a36f496fbb75462f500f4035" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 42805 } } peers { permanent_uuid: "41789aa446ae4c5bbcda631441a01566" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 41977 } } peers { permanent_uuid: "b9807e3fbedb4973a065b0c81113fa6a" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 41849 } }
I20260620 09:26:12.149554 18800 leader_election.cc:288] T f3f67960ea2f4393902c37160f338873 P b9807e3fbedb4973a065b0c81113fa6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9e86ec40a36f496fbb75462f500f4035 (127.16.43.129:42805), 41789aa446ae4c5bbcda631441a01566 (127.16.43.130:41977)
I20260620 09:26:12.153256 18791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:12.149953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.131:36381 (local address 127.16.43.129:42805)
0620 09:26:12.149996 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:12.149998 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:12.150063 (+ 65us) server_negotiation.cc:407] Connection header received
0620 09:26:12.150130 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:12.150131 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:12.150141 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:12.150189 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:26:12.150731 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:12.151302 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:12.152280 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:12.152825 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:12.152904 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:12.152914 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:12.153123 (+ 209us) server_negotiation.cc:299] Negotiation successful
0620 09:26:12.153175 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 09:26:12.153296 18792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:12.149902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.131:44443 (local address 127.16.43.130:41977)
0620 09:26:12.150030 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:12.150032 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:12.150040 (+ 8us) server_negotiation.cc:407] Connection header received
0620 09:26:12.150071 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:12.150071 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:12.150081 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:12.150156 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:26:12.150325 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:12.150902 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:12.152336 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:12.152900 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:12.152967 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:12.152973 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:12.153198 (+ 225us) server_negotiation.cc:299] Negotiation successful
0620 09:26:12.153235 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260620 09:26:12.153296 18665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:12.149770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.130:41977 (local address 127.16.43.131:44443)
0620 09:26:12.149813 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:12.149826 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:12.149870 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:12.150160 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:12.150161 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:12.150162 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:26:12.150238 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:12.150240 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:12.150997 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:12.150998 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:12.152258 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:12.152260 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:12.153001 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:12.153002 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:12.153103 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:12.153107 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 09:26:12.153178 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 09:26:12.153219 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260620 09:26:12.153467 18841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:12.149770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.129:42805 (local address 127.16.43.131:36381)
0620 09:26:12.149976 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:12.149986 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:12.150072 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:12.150228 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:12.150229 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:12.150232 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:26:12.150631 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:12.150633 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:12.151386 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:12.151387 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:12.152183 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:12.152186 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:12.152921 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:12.152922 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:12.153000 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:12.153004 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 09:26:12.153342 (+ 338us) client_negotiation.cc:240] Negotiation successful
0620 09:26:12.153385 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"mutex_wait_us":48,"thread_start_us":63,"threads_started":1}
I20260620 09:26:12.153640 18586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3f67960ea2f4393902c37160f338873" candidate_uuid: "b9807e3fbedb4973a065b0c81113fa6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41789aa446ae4c5bbcda631441a01566" is_pre_election: true
I20260620 09:26:12.153757 18803 raft_consensus.cc:482] T f3f67960ea2f4393902c37160f338873 P 9e86ec40a36f496fbb75462f500f4035 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:26:12.153786 18586 raft_consensus.cc:2454] T f3f67960ea2f4393902c37160f338873 P 41789aa446ae4c5bbcda631441a01566 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9807e3fbedb4973a065b0c81113fa6a in term 0.
I20260620 09:26:12.153811 18803 raft_consensus.cc:504] T f3f67960ea2f4393902c37160f338873 P 9e86ec40a36f496fbb75462f500f4035 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e86ec40a36f496fbb75462f500f4035" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 42805 } } peers { permanent_uuid: "41789aa446ae4c5bbcda631441a01566" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 41977 } } peers { permanent_uuid: "b9807e3fbedb4973a065b0c81113fa6a" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 41849 } }
I20260620 09:26:12.153909 18446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3f67960ea2f4393902c37160f338873" candidate_uuid: "b9807e3fbedb4973a065b0c81113fa6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e86ec40a36f496fbb75462f500f4035" is_pre_election: true
I20260620 09:26:12.153987 18446 raft_consensus.cc:2454] T f3f67960ea2f4393902c37160f338873 P 9e86ec40a36f496fbb75462f500f4035 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9807e3fbedb4973a065b0c81113fa6a in term 0.
I20260620 09:26:12.153959 18659 leader_election.cc:302] T f3f67960ea2f4393902c37160f338873 P b9807e3fbedb4973a065b0c81113fa6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41789aa446ae4c5bbcda631441a01566, b9807e3fbedb4973a065b0c81113fa6a; no voters:
I20260620 09:26:12.154076 18800 raft_consensus.cc:2781] T f3f67960ea2f4393902c37160f338873 P b9807e3fbedb4973a065b0c81113fa6a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 09:26:12.154112 18800 raft_consensus.cc:482] T f3f67960ea2f4393902c37160f338873 P b9807e3fbedb4973a065b0c81113fa6a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 09:26:12.154134 18800 raft_consensus.cc:3037] T f3f67960ea2f4393902c37160f338873 P b9807e3fbedb4973a065b0c81113fa6a [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:26:12.154124 18803 leader_election.cc:288] T f3f67960ea2f4393902c37160f338873 P 9e86ec40a36f496fbb75462f500f4035 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 41789aa446ae4c5bbcda631441a01566 (127.16.43.130:41977), b9807e3fbedb4973a065b0c81113fa6a (127.16.43.131:41849)
I20260620 09:26:12.155030 18800 raft_consensus.cc:504] T f3f67960ea2f4393902c37160f338873 P b9807e3fbedb4973a065b0c81113fa6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e86ec40a36f496fbb75462f500f4035" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 42805 } } peers { permanent_uuid: "41789aa446ae4c5bbcda631441a01566" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 41977 } } peers { permanent_uuid: "b9807e3fbedb4973a065b0c81113fa6a" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 41849 } }
I20260620 09:26:12.155165 18800 leader_election.cc:288] T f3f67960ea2f4393902c37160f338873 P b9807e3fbedb4973a065b0c81113fa6a [CANDIDATE]: Term 1 election: Requested vote from peers 9e86ec40a36f496fbb75462f500f4035 (127.16.43.129:42805), 41789aa446ae4c5bbcda631441a01566 (127.16.43.130:41977)
I20260620 09:26:12.155360 18446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3f67960ea2f4393902c37160f338873" candidate_uuid: "b9807e3fbedb4973a065b0c81113fa6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e86ec40a36f496fbb75462f500f4035"
I20260620 09:26:12.155416 18446 raft_consensus.cc:3037] T f3f67960ea2f4393902c37160f338873 P 9e86ec40a36f496fbb75462f500f4035 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:26:12.155781 18586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3f67960ea2f4393902c37160f338873" candidate_uuid: "b9807e3fbedb4973a065b0c81113fa6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41789aa446ae4c5bbcda631441a01566"
I20260620 09:26:12.155866 18586 raft_consensus.cc:3037] T f3f67960ea2f4393902c37160f338873 P 41789aa446ae4c5bbcda631441a01566 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:26:12.156180 18446 raft_consensus.cc:2454] T f3f67960ea2f4393902c37160f338873 P 9e86ec40a36f496fbb75462f500f4035 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9807e3fbedb4973a065b0c81113fa6a in term 1.
I20260620 09:26:12.156349 18658 leader_election.cc:302] T f3f67960ea2f4393902c37160f338873 P b9807e3fbedb4973a065b0c81113fa6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9e86ec40a36f496fbb75462f500f4035, b9807e3fbedb4973a065b0c81113fa6a; no voters:
I20260620 09:26:12.156435 18800 raft_consensus.cc:2781] T f3f67960ea2f4393902c37160f338873 P b9807e3fbedb4973a065b0c81113fa6a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:26:12.156550 18800 raft_consensus.cc:686] T f3f67960ea2f4393902c37160f338873 P b9807e3fbedb4973a065b0c81113fa6a [term 1 LEADER]: Becoming Leader. State: Replica: b9807e3fbedb4973a065b0c81113fa6a, State: Running, Role: LEADER
I20260620 09:26:12.156632 18800 consensus_queue.cc:237] T f3f67960ea2f4393902c37160f338873 P b9807e3fbedb4973a065b0c81113fa6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e86ec40a36f496fbb75462f500f4035" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 42805 } } peers { permanent_uuid: "41789aa446ae4c5bbcda631441a01566" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 41977 } } peers { permanent_uuid: "b9807e3fbedb4973a065b0c81113fa6a" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 41849 } }
I20260620 09:26:12.156736 18586 raft_consensus.cc:2454] T f3f67960ea2f4393902c37160f338873 P 41789aa446ae4c5bbcda631441a01566 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9807e3fbedb4973a065b0c81113fa6a in term 1.
I20260620 09:26:12.157311 18287 catalog_manager.cc:5495] T f3f67960ea2f4393902c37160f338873 P b9807e3fbedb4973a065b0c81113fa6a reported cstate change: term changed from 0 to 1, leader changed from <none> to b9807e3fbedb4973a065b0c81113fa6a (127.16.43.131). New cstate: current_term: 1 leader_uuid: "b9807e3fbedb4973a065b0c81113fa6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e86ec40a36f496fbb75462f500f4035" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 42805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "41789aa446ae4c5bbcda631441a01566" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 41977 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9807e3fbedb4973a065b0c81113fa6a" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 41849 } health_report { overall_health: HEALTHY } } }
I20260620 09:26:12.159195 18792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:12.154529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.129:60327 (local address 127.16.43.130:41977)
0620 09:26:12.154581 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:12.154583 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:12.154638 (+ 55us) server_negotiation.cc:407] Connection header received
0620 09:26:12.154738 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:12.154739 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:12.154748 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:12.154773 (+ 25us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:26:12.154951 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:12.155596 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:12.158345 (+ 2749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:12.158861 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:12.158944 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:12.158952 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:12.159100 (+ 148us) server_negotiation.cc:299] Negotiation successful
0620 09:26:12.159133 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 09:26:12.159214 18843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:12.154268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.130:41977 (local address 127.16.43.129:60327)
0620 09:26:12.154572 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:12.154584 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:12.154663 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:12.154829 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:12.154830 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:12.154831 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:26:12.154879 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:12.154880 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:12.157042 (+ 2162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:12.157043 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:12.158259 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:12.158262 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:12.158954 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:12.158955 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:12.159027 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:12.159032 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 09:26:12.159093 (+ 61us) client_negotiation.cc:240] Negotiation successful
0620 09:26:12.159133 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":54,"threads_started":1}
I20260620 09:26:12.159476 18794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:12.155328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.129:38585 (local address 127.16.43.131:41849)
0620 09:26:12.155362 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:12.155364 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:12.155370 (+ 6us) server_negotiation.cc:407] Connection header received
0620 09:26:12.155388 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:12.155389 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:12.155396 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:12.155440 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:26:12.155741 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:12.156352 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:12.158570 (+ 2218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:12.159116 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:12.159185 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:12.159193 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:12.159367 (+ 174us) server_negotiation.cc:299] Negotiation successful
0620 09:26:12.159415 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 09:26:12.159483 18842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:12.154417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.131:41849 (local address 127.16.43.129:38585)
0620 09:26:12.154617 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:12.154625 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:12.154663 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:12.155565 (+ 902us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:12.155566 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:12.155568 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:26:12.155635 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:12.155637 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:12.156556 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:12.156557 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:12.158482 (+ 1925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:12.158484 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:12.159226 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:12.159227 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:12.159301 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:12.159305 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 09:26:12.159359 (+ 54us) client_negotiation.cc:240] Negotiation successful
0620 09:26:12.159398 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"mutex_wait_us":1,"thread_start_us":43,"threads_started":1}
I20260620 09:26:12.159781 18586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3f67960ea2f4393902c37160f338873" candidate_uuid: "9e86ec40a36f496fbb75462f500f4035" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41789aa446ae4c5bbcda631441a01566" is_pre_election: true
I20260620 09:26:12.159796 18726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3f67960ea2f4393902c37160f338873" candidate_uuid: "9e86ec40a36f496fbb75462f500f4035" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9807e3fbedb4973a065b0c81113fa6a" is_pre_election: true
I20260620 09:26:12.159901 18586 raft_consensus.cc:2379] T f3f67960ea2f4393902c37160f338873 P 41789aa446ae4c5bbcda631441a01566 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9e86ec40a36f496fbb75462f500f4035 in current term 1: Already voted for candidate b9807e3fbedb4973a065b0c81113fa6a in this term.
I20260620 09:26:12.160205 18379 leader_election.cc:302] T f3f67960ea2f4393902c37160f338873 P 9e86ec40a36f496fbb75462f500f4035 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9e86ec40a36f496fbb75462f500f4035; no voters: 41789aa446ae4c5bbcda631441a01566, b9807e3fbedb4973a065b0c81113fa6a
I20260620 09:26:12.160377 18803 raft_consensus.cc:2726] T f3f67960ea2f4393902c37160f338873 P 9e86ec40a36f496fbb75462f500f4035 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260620 09:26:12.168035 18794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:12.165161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52338 (local address 127.16.43.131:41849)
0620 09:26:12.165215 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:12.165217 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:12.165224 (+ 7us) server_negotiation.cc:407] Connection header received
0620 09:26:12.165257 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:12.165257 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:12.165279 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:12.165364 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:26:12.165482 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:12.166219 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:12.167123 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:12.167387 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:12.167480 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:12.167676 (+ 196us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:26:12.167800 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:26:12.167891 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:12.167925 (+ 34us) server_negotiation.cc:299] Negotiation successful
0620 09:26:12.167949 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 09:26:12.173024 18792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:12.170543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.131:33925 (local address 127.16.43.130:41977)
0620 09:26:12.170594 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:12.170596 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:12.170605 (+ 9us) server_negotiation.cc:407] Connection header received
0620 09:26:12.170664 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:12.170665 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:12.170676 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:12.170724 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:26:12.170852 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:12.171384 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:12.172219 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:12.172473 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:12.172548 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:12.172738 (+ 190us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:26:12.172773 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:26:12.172855 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:12.172917 (+ 62us) server_negotiation.cc:299] Negotiation successful
0620 09:26:12.172938 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 09:26:12.173024 18778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:12.170469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.130:41977 (local address 127.16.43.131:33925)
0620 09:26:12.170543 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:12.170554 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:12.170598 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:12.170724 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:12.170725 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:12.170726 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 09:26:12.170782 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:12.170784 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:12.171465 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:12.171466 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:12.172129 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:12.172132 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:12.172562 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:12.172563 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:12.172630 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:12.172643 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 09:26:12.172857 (+ 214us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 09:26:12.172860 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 09:26:12.172912 (+ 52us) client_negotiation.cc:240] Negotiation successful
0620 09:26:12.172958 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260620 09:26:12.175195 18446 raft_consensus.cc:1261] T f3f67960ea2f4393902c37160f338873 P 9e86ec40a36f496fbb75462f500f4035 [term 1 FOLLOWER]: Refusing update from remote peer b9807e3fbedb4973a065b0c81113fa6a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:26:12.175293 18586 raft_consensus.cc:1261] T f3f67960ea2f4393902c37160f338873 P 41789aa446ae4c5bbcda631441a01566 [term 1 FOLLOWER]: Refusing update from remote peer b9807e3fbedb4973a065b0c81113fa6a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:26:12.175495 18800 consensus_queue.cc:1035] T f3f67960ea2f4393902c37160f338873 P b9807e3fbedb4973a065b0c81113fa6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "41789aa446ae4c5bbcda631441a01566" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 41977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:26:12.175760 18844 consensus_queue.cc:1035] T f3f67960ea2f4393902c37160f338873 P b9807e3fbedb4973a065b0c81113fa6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e86ec40a36f496fbb75462f500f4035" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 42805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:26:12.183413 18794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:12.180979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.130:39521 (local address 127.16.43.131:41849)
0620 09:26:12.181017 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:12.181019 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:12.181028 (+ 9us) server_negotiation.cc:407] Connection header received
0620 09:26:12.181049 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:12.181049 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:12.181059 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:12.181113 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:26:12.181233 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:12.181965 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:12.182714 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:12.183004 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:12.183070 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:12.183230 (+ 160us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:26:12.183258 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:26:12.183299 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:12.183351 (+ 52us) server_negotiation.cc:299] Negotiation successful
0620 09:26:12.183365 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 09:26:12.183470 18638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:12.180668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.131:41849 (local address 127.16.43.130:39521)
0620 09:26:12.180745 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:12.180756 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:12.180791 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:12.181115 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:12.181116 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:12.181117 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 09:26:12.181159 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:12.181160 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:12.182057 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:12.182058 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:12.182625 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:12.182627 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:12.183104 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:12.183105 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:12.183166 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:12.183170 (+ 4us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 09:26:12.183299 (+ 129us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 09:26:12.183301 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 09:26:12.183349 (+ 48us) client_negotiation.cc:240] Negotiation successful
0620 09:26:12.183394 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
W20260620 09:26:12.183759 18822 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 09:26:13.702890 18287 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56554:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:26:13.702994 18287 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56554:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:26:13.703872 18287 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 21cc679fd523463da96d6adcab8040db: Sending DeleteTablet for 3 replicas of tablet f3f67960ea2f4393902c37160f338873
I20260620 09:26:13.704211 18426 tablet_service.cc:1514] Processing DeleteTablet for tablet f3f67960ea2f4393902c37160f338873 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:26:13 UTC) from {username='kudu', principal='kudu/127.16.43.190@KRBTEST.COM'} at 127.0.0.1:53164
I20260620 09:26:13.704244 18706 tablet_service.cc:1514] Processing DeleteTablet for tablet f3f67960ea2f4393902c37160f338873 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:26:13 UTC) from {username='kudu', principal='kudu/127.16.43.190@KRBTEST.COM'} at 127.0.0.1:52330
I20260620 09:26:13.704411 18565 tablet_service.cc:1514] Processing DeleteTablet for tablet f3f67960ea2f4393902c37160f338873 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:26:13 UTC) from {username='kudu', principal='kudu/127.16.43.190@KRBTEST.COM'} at 127.0.0.1:53952
I20260620 09:26:13.704432 18874 tablet_replica.cc:330] T f3f67960ea2f4393902c37160f338873 P b9807e3fbedb4973a065b0c81113fa6a: stopping tablet replica
I20260620 09:26:13.704491 18874 raft_consensus.cc:2229] T f3f67960ea2f4393902c37160f338873 P b9807e3fbedb4973a065b0c81113fa6a [term 1 LEADER]: Raft consensus shutting down.
I20260620 09:26:13.704632 18874 raft_consensus.cc:2258] T f3f67960ea2f4393902c37160f338873 P b9807e3fbedb4973a065b0c81113fa6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:26:13.704625 18875 tablet_replica.cc:330] T f3f67960ea2f4393902c37160f338873 P 41789aa446ae4c5bbcda631441a01566: stopping tablet replica
I20260620 09:26:13.704731 18875 raft_consensus.cc:2229] T f3f67960ea2f4393902c37160f338873 P 41789aa446ae4c5bbcda631441a01566 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:26:13.704774 18875 raft_consensus.cc:2258] T f3f67960ea2f4393902c37160f338873 P 41789aa446ae4c5bbcda631441a01566 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:26:13.705010 18875 ts_tablet_manager.cc:1905] T f3f67960ea2f4393902c37160f338873 P 41789aa446ae4c5bbcda631441a01566: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:26:13.704999 18874 ts_tablet_manager.cc:1905] T f3f67960ea2f4393902c37160f338873 P b9807e3fbedb4973a065b0c81113fa6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:26:13.707070 18875 ts_tablet_manager.cc:1918] T f3f67960ea2f4393902c37160f338873 P 41789aa446ae4c5bbcda631441a01566: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 09:26:13.707082 18874 ts_tablet_manager.cc:1918] T f3f67960ea2f4393902c37160f338873 P b9807e3fbedb4973a065b0c81113fa6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 09:26:13.707145 18874 log.cc:1196] T f3f67960ea2f4393902c37160f338873 P b9807e3fbedb4973a065b0c81113fa6a: Deleting WAL directory at /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781947562949448-16558-0/minicluster-data/ts-2/wal/wals/f3f67960ea2f4393902c37160f338873
I20260620 09:26:13.707159 18873 tablet_replica.cc:330] T f3f67960ea2f4393902c37160f338873 P 9e86ec40a36f496fbb75462f500f4035: stopping tablet replica
I20260620 09:26:13.707185 18875 log.cc:1196] T f3f67960ea2f4393902c37160f338873 P 41789aa446ae4c5bbcda631441a01566: Deleting WAL directory at /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781947562949448-16558-0/minicluster-data/ts-1/wal/wals/f3f67960ea2f4393902c37160f338873
I20260620 09:26:13.707217 18873 raft_consensus.cc:2229] T f3f67960ea2f4393902c37160f338873 P 9e86ec40a36f496fbb75462f500f4035 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:26:13.707288 18873 raft_consensus.cc:2258] T f3f67960ea2f4393902c37160f338873 P 9e86ec40a36f496fbb75462f500f4035 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:26:13.707423 18874 ts_tablet_manager.cc:1939] T f3f67960ea2f4393902c37160f338873 P b9807e3fbedb4973a065b0c81113fa6a: Deleting consensus metadata
I20260620 09:26:13.707463 18875 ts_tablet_manager.cc:1939] T f3f67960ea2f4393902c37160f338873 P 41789aa446ae4c5bbcda631441a01566: Deleting consensus metadata
I20260620 09:26:13.707644 18873 ts_tablet_manager.cc:1905] T f3f67960ea2f4393902c37160f338873 P 9e86ec40a36f496fbb75462f500f4035: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:26:13.707882 18275 catalog_manager.cc:4841] TS 41789aa446ae4c5bbcda631441a01566 (127.16.43.130:41977): tablet f3f67960ea2f4393902c37160f338873 (table test-table [id=19c71355bd0b4886a4580159543ba8bf]) successfully deleted
I20260620 09:26:13.707960 18275 catalog_manager.cc:4841] TS b9807e3fbedb4973a065b0c81113fa6a (127.16.43.131:41849): tablet f3f67960ea2f4393902c37160f338873 (table test-table [id=19c71355bd0b4886a4580159543ba8bf]) successfully deleted
I20260620 09:26:13.709084 18873 ts_tablet_manager.cc:1918] T f3f67960ea2f4393902c37160f338873 P 9e86ec40a36f496fbb75462f500f4035: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 09:26:13.709161 18873 log.cc:1196] T f3f67960ea2f4393902c37160f338873 P 9e86ec40a36f496fbb75462f500f4035: Deleting WAL directory at /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781947562949448-16558-0/minicluster-data/ts-0/wal/wals/f3f67960ea2f4393902c37160f338873
I20260620 09:26:13.709383 18873 ts_tablet_manager.cc:1939] T f3f67960ea2f4393902c37160f338873 P 9e86ec40a36f496fbb75462f500f4035: Deleting consensus metadata
I20260620 09:26:13.709731 18274 catalog_manager.cc:4841] TS 9e86ec40a36f496fbb75462f500f4035 (127.16.43.129:42805): tablet f3f67960ea2f4393902c37160f338873 (table test-table [id=19c71355bd0b4886a4580159543ba8bf]) successfully deleted
Jun 20 09:26:13 dist-test-slave-706q krb5kdc[18244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947573, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 09:26:13 dist-test-slave-706q krb5kdc[18244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947573, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.43.190@KRBTEST.COM
I20260620 09:26:13.731513 18883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:13.723169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56572 (local address 127.16.43.190:42057)
0620 09:26:13.723329 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:13.723331 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:13.723346 (+ 15us) server_negotiation.cc:407] Connection header received
0620 09:26:13.723393 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:13.723394 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:13.723409 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:13.723482 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:13.724254 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:13.724878 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:13.725724 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:13.725931 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:13.726008 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:13.728575 (+ 2567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:13.728577 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:13.728577 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:26:13.728586 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:13.730140 (+ 1554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:13.730501 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:13.730503 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:13.730503 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:13.730586 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:13.730778 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:13.730778 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:13.730778 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:13.730905 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:13.730999 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:13.731335 (+ 336us) server_negotiation.cc:299] Negotiation successful
0620 09:26:13.731416 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":106,"threads_started":1}
I20260620 09:26:13.733312 18287 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56572:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 09:26:13.733465 18287 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 09:26:13.734936 18426 tablet_service.cc:1467] Processing CreateTablet for tablet 95bbd4529955431ca482f0d4bdbadc85 (DEFAULT_TABLE table=test-table [id=ac69f708b390417babb362478e86cb49]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:26:13.735095 18426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95bbd4529955431ca482f0d4bdbadc85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:26:13.735707 18706 tablet_service.cc:1467] Processing CreateTablet for tablet 95bbd4529955431ca482f0d4bdbadc85 (DEFAULT_TABLE table=test-table [id=ac69f708b390417babb362478e86cb49]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:26:13.735819 18706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95bbd4529955431ca482f0d4bdbadc85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:26:13.736553 18884 tablet_bootstrap.cc:492] T 95bbd4529955431ca482f0d4bdbadc85 P 9e86ec40a36f496fbb75462f500f4035: Bootstrap starting.
I20260620 09:26:13.736935 18884 tablet_bootstrap.cc:654] T 95bbd4529955431ca482f0d4bdbadc85 P 9e86ec40a36f496fbb75462f500f4035: Neither blocks nor log segments found. Creating new log.
I20260620 09:26:13.737370 18886 tablet_bootstrap.cc:492] T 95bbd4529955431ca482f0d4bdbadc85 P b9807e3fbedb4973a065b0c81113fa6a: Bootstrap starting.
I20260620 09:26:13.737432 18565 tablet_service.cc:1467] Processing CreateTablet for tablet 95bbd4529955431ca482f0d4bdbadc85 (DEFAULT_TABLE table=test-table [id=ac69f708b390417babb362478e86cb49]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:26:13.737547 18565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95bbd4529955431ca482f0d4bdbadc85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:26:13.737548 18884 tablet_bootstrap.cc:492] T 95bbd4529955431ca482f0d4bdbadc85 P 9e86ec40a36f496fbb75462f500f4035: No bootstrap required, opened a new log
I20260620 09:26:13.737594 18884 ts_tablet_manager.cc:1397] T 95bbd4529955431ca482f0d4bdbadc85 P 9e86ec40a36f496fbb75462f500f4035: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 09:26:13.737715 18884 raft_consensus.cc:348] T 95bbd4529955431ca482f0d4bdbadc85 P 9e86ec40a36f496fbb75462f500f4035 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e86ec40a36f496fbb75462f500f4035" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 42805 } } peers { permanent_uuid: "41789aa446ae4c5bbcda631441a01566" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 41977 } } peers { permanent_uuid: "b9807e3fbedb4973a065b0c81113fa6a" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 41849 } }
I20260620 09:26:13.737764 18884 raft_consensus.cc:374] T 95bbd4529955431ca482f0d4bdbadc85 P 9e86ec40a36f496fbb75462f500f4035 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:26:13.737777 18884 raft_consensus.cc:729] T 95bbd4529955431ca482f0d4bdbadc85 P 9e86ec40a36f496fbb75462f500f4035 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e86ec40a36f496fbb75462f500f4035, State: Initialized, Role: FOLLOWER
I20260620 09:26:13.737845 18886 tablet_bootstrap.cc:654] T 95bbd4529955431ca482f0d4bdbadc85 P b9807e3fbedb4973a065b0c81113fa6a: Neither blocks nor log segments found. Creating new log.
I20260620 09:26:13.737838 18884 consensus_queue.cc:260] T 95bbd4529955431ca482f0d4bdbadc85 P 9e86ec40a36f496fbb75462f500f4035 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e86ec40a36f496fbb75462f500f4035" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 42805 } } peers { permanent_uuid: "41789aa446ae4c5bbcda631441a01566" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 41977 } } peers { permanent_uuid: "b9807e3fbedb4973a065b0c81113fa6a" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 41849 } }
I20260620 09:26:13.738039 18884 ts_tablet_manager.cc:1428] T 95bbd4529955431ca482f0d4bdbadc85 P 9e86ec40a36f496fbb75462f500f4035: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 09:26:13.739009 18889 tablet_bootstrap.cc:492] T 95bbd4529955431ca482f0d4bdbadc85 P 41789aa446ae4c5bbcda631441a01566: Bootstrap starting.
I20260620 09:26:13.739332 18889 tablet_bootstrap.cc:654] T 95bbd4529955431ca482f0d4bdbadc85 P 41789aa446ae4c5bbcda631441a01566: Neither blocks nor log segments found. Creating new log.
I20260620 09:26:13.739578 18886 tablet_bootstrap.cc:492] T 95bbd4529955431ca482f0d4bdbadc85 P b9807e3fbedb4973a065b0c81113fa6a: No bootstrap required, opened a new log
I20260620 09:26:13.739666 18886 ts_tablet_manager.cc:1397] T 95bbd4529955431ca482f0d4bdbadc85 P b9807e3fbedb4973a065b0c81113fa6a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 09:26:13.739775 18886 raft_consensus.cc:348] T 95bbd4529955431ca482f0d4bdbadc85 P b9807e3fbedb4973a065b0c81113fa6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e86ec40a36f496fbb75462f500f4035" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 42805 } } peers { permanent_uuid: "41789aa446ae4c5bbcda631441a01566" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 41977 } } peers { permanent_uuid: "b9807e3fbedb4973a065b0c81113fa6a" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 41849 } }
I20260620 09:26:13.739832 18886 raft_consensus.cc:374] T 95bbd4529955431ca482f0d4bdbadc85 P b9807e3fbedb4973a065b0c81113fa6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:26:13.739845 18886 raft_consensus.cc:729] T 95bbd4529955431ca482f0d4bdbadc85 P b9807e3fbedb4973a065b0c81113fa6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9807e3fbedb4973a065b0c81113fa6a, State: Initialized, Role: FOLLOWER
I20260620 09:26:13.739908 18886 consensus_queue.cc:260] T 95bbd4529955431ca482f0d4bdbadc85 P b9807e3fbedb4973a065b0c81113fa6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e86ec40a36f496fbb75462f500f4035" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 42805 } } peers { permanent_uuid: "41789aa446ae4c5bbcda631441a01566" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 41977 } } peers { permanent_uuid: "b9807e3fbedb4973a065b0c81113fa6a" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 41849 } }
I20260620 09:26:13.740010 18886 ts_tablet_manager.cc:1428] T 95bbd4529955431ca482f0d4bdbadc85 P b9807e3fbedb4973a065b0c81113fa6a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:13.740439 18889 tablet_bootstrap.cc:492] T 95bbd4529955431ca482f0d4bdbadc85 P 41789aa446ae4c5bbcda631441a01566: No bootstrap required, opened a new log
I20260620 09:26:13.740499 18889 ts_tablet_manager.cc:1397] T 95bbd4529955431ca482f0d4bdbadc85 P 41789aa446ae4c5bbcda631441a01566: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 09:26:13.740615 18889 raft_consensus.cc:348] T 95bbd4529955431ca482f0d4bdbadc85 P 41789aa446ae4c5bbcda631441a01566 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e86ec40a36f496fbb75462f500f4035" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 42805 } } peers { permanent_uuid: "41789aa446ae4c5bbcda631441a01566" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 41977 } } peers { permanent_uuid: "b9807e3fbedb4973a065b0c81113fa6a" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 41849 } }
I20260620 09:26:13.740698 18889 raft_consensus.cc:374] T 95bbd4529955431ca482f0d4bdbadc85 P 41789aa446ae4c5bbcda631441a01566 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:26:13.740731 18889 raft_consensus.cc:729] T 95bbd4529955431ca482f0d4bdbadc85 P 41789aa446ae4c5bbcda631441a01566 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41789aa446ae4c5bbcda631441a01566, State: Initialized, Role: FOLLOWER
I20260620 09:26:13.740804 18889 consensus_queue.cc:260] T 95bbd4529955431ca482f0d4bdbadc85 P 41789aa446ae4c5bbcda631441a01566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e86ec40a36f496fbb75462f500f4035" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 42805 } } peers { permanent_uuid: "41789aa446ae4c5bbcda631441a01566" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 41977 } } peers { permanent_uuid: "b9807e3fbedb4973a065b0c81113fa6a" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 41849 } }
I20260620 09:26:13.740985 18889 ts_tablet_manager.cc:1428] T 95bbd4529955431ca482f0d4bdbadc85 P 41789aa446ae4c5bbcda631441a01566: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260620 09:26:13.769198 18887 raft_consensus.cc:482] T 95bbd4529955431ca482f0d4bdbadc85 P 9e86ec40a36f496fbb75462f500f4035 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:26:13.769308 18887 raft_consensus.cc:504] T 95bbd4529955431ca482f0d4bdbadc85 P 9e86ec40a36f496fbb75462f500f4035 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e86ec40a36f496fbb75462f500f4035" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 42805 } } peers { permanent_uuid: "41789aa446ae4c5bbcda631441a01566" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 41977 } } peers { permanent_uuid: "b9807e3fbedb4973a065b0c81113fa6a" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 41849 } }
I20260620 09:26:13.769445 18887 leader_election.cc:288] T 95bbd4529955431ca482f0d4bdbadc85 P 9e86ec40a36f496fbb75462f500f4035 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 41789aa446ae4c5bbcda631441a01566 (127.16.43.130:41977), b9807e3fbedb4973a065b0c81113fa6a (127.16.43.131:41849)
I20260620 09:26:13.769716 18586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95bbd4529955431ca482f0d4bdbadc85" candidate_uuid: "9e86ec40a36f496fbb75462f500f4035" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41789aa446ae4c5bbcda631441a01566" is_pre_election: true
I20260620 09:26:13.769747 18726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95bbd4529955431ca482f0d4bdbadc85" candidate_uuid: "9e86ec40a36f496fbb75462f500f4035" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9807e3fbedb4973a065b0c81113fa6a" is_pre_election: true
I20260620 09:26:13.769829 18586 raft_consensus.cc:2454] T 95bbd4529955431ca482f0d4bdbadc85 P 41789aa446ae4c5bbcda631441a01566 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e86ec40a36f496fbb75462f500f4035 in term 0.
I20260620 09:26:13.769833 18726 raft_consensus.cc:2454] T 95bbd4529955431ca482f0d4bdbadc85 P b9807e3fbedb4973a065b0c81113fa6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e86ec40a36f496fbb75462f500f4035 in term 0.
I20260620 09:26:13.770026 18379 leader_election.cc:302] T 95bbd4529955431ca482f0d4bdbadc85 P 9e86ec40a36f496fbb75462f500f4035 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9e86ec40a36f496fbb75462f500f4035, b9807e3fbedb4973a065b0c81113fa6a; no voters:
I20260620 09:26:13.770164 18887 raft_consensus.cc:2781] T 95bbd4529955431ca482f0d4bdbadc85 P 9e86ec40a36f496fbb75462f500f4035 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 09:26:13.770206 18887 raft_consensus.cc:482] T 95bbd4529955431ca482f0d4bdbadc85 P 9e86ec40a36f496fbb75462f500f4035 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 09:26:13.770226 18887 raft_consensus.cc:3037] T 95bbd4529955431ca482f0d4bdbadc85 P 9e86ec40a36f496fbb75462f500f4035 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:26:13.770990 18887 raft_consensus.cc:504] T 95bbd4529955431ca482f0d4bdbadc85 P 9e86ec40a36f496fbb75462f500f4035 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e86ec40a36f496fbb75462f500f4035" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 42805 } } peers { permanent_uuid: "41789aa446ae4c5bbcda631441a01566" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 41977 } } peers { permanent_uuid: "b9807e3fbedb4973a065b0c81113fa6a" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 41849 } }
I20260620 09:26:13.771142 18887 leader_election.cc:288] T 95bbd4529955431ca482f0d4bdbadc85 P 9e86ec40a36f496fbb75462f500f4035 [CANDIDATE]: Term 1 election: Requested vote from peers 41789aa446ae4c5bbcda631441a01566 (127.16.43.130:41977), b9807e3fbedb4973a065b0c81113fa6a (127.16.43.131:41849)
I20260620 09:26:13.771327 18586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95bbd4529955431ca482f0d4bdbadc85" candidate_uuid: "9e86ec40a36f496fbb75462f500f4035" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41789aa446ae4c5bbcda631441a01566"
I20260620 09:26:13.771415 18586 raft_consensus.cc:3037] T 95bbd4529955431ca482f0d4bdbadc85 P 41789aa446ae4c5bbcda631441a01566 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:26:13.771391 18726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95bbd4529955431ca482f0d4bdbadc85" candidate_uuid: "9e86ec40a36f496fbb75462f500f4035" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9807e3fbedb4973a065b0c81113fa6a"
I20260620 09:26:13.771464 18726 raft_consensus.cc:3037] T 95bbd4529955431ca482f0d4bdbadc85 P b9807e3fbedb4973a065b0c81113fa6a [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:26:13.772305 18726 raft_consensus.cc:2454] T 95bbd4529955431ca482f0d4bdbadc85 P b9807e3fbedb4973a065b0c81113fa6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e86ec40a36f496fbb75462f500f4035 in term 1.
I20260620 09:26:13.772305 18586 raft_consensus.cc:2454] T 95bbd4529955431ca482f0d4bdbadc85 P 41789aa446ae4c5bbcda631441a01566 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e86ec40a36f496fbb75462f500f4035 in term 1.
I20260620 09:26:13.772483 18379 leader_election.cc:302] T 95bbd4529955431ca482f0d4bdbadc85 P 9e86ec40a36f496fbb75462f500f4035 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9e86ec40a36f496fbb75462f500f4035, b9807e3fbedb4973a065b0c81113fa6a; no voters:
I20260620 09:26:13.772637 18887 raft_consensus.cc:2781] T 95bbd4529955431ca482f0d4bdbadc85 P 9e86ec40a36f496fbb75462f500f4035 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:26:13.772827 18887 raft_consensus.cc:686] T 95bbd4529955431ca482f0d4bdbadc85 P 9e86ec40a36f496fbb75462f500f4035 [term 1 LEADER]: Becoming Leader. State: Replica: 9e86ec40a36f496fbb75462f500f4035, State: Running, Role: LEADER
I20260620 09:26:13.772925 18887 consensus_queue.cc:237] T 95bbd4529955431ca482f0d4bdbadc85 P 9e86ec40a36f496fbb75462f500f4035 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e86ec40a36f496fbb75462f500f4035" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 42805 } } peers { permanent_uuid: "41789aa446ae4c5bbcda631441a01566" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 41977 } } peers { permanent_uuid: "b9807e3fbedb4973a065b0c81113fa6a" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 41849 } }
I20260620 09:26:13.773629 18287 catalog_manager.cc:5495] T 95bbd4529955431ca482f0d4bdbadc85 P 9e86ec40a36f496fbb75462f500f4035 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9e86ec40a36f496fbb75462f500f4035 (127.16.43.129). New cstate: current_term: 1 leader_uuid: "9e86ec40a36f496fbb75462f500f4035" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e86ec40a36f496fbb75462f500f4035" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 42805 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "41789aa446ae4c5bbcda631441a01566" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 41977 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9807e3fbedb4973a065b0c81113fa6a" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 41849 } health_report { overall_health: UNKNOWN } } }
I20260620 09:26:13.788213 18894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:13.785525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53172 (local address 127.16.43.129:42805)
0620 09:26:13.785711 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:13.785713 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:13.785719 (+ 6us) server_negotiation.cc:407] Connection header received
0620 09:26:13.785752 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:13.785753 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:13.785771 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:13.785872 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:26:13.785994 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:13.786560 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:13.787366 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:13.787647 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:13.787722 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:13.787860 (+ 138us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:26:13.787983 (+ 123us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:26:13.788071 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:13.788093 (+ 22us) server_negotiation.cc:299] Negotiation successful
0620 09:26:13.788116 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":142,"threads_started":1}
I20260620 09:26:13.793517 18897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:13.790833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.130:41977 (local address 127.16.43.129:44149)
0620 09:26:13.791006 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:13.791055 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:13.791115 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:13.791282 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:13.791283 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:13.791284 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 09:26:13.791330 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:13.791331 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:13.792064 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:13.792065 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:13.792651 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:13.792653 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:13.793055 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:13.793057 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:13.793110 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:13.793132 (+ 22us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 09:26:13.793333 (+ 201us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 09:26:13.793336 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 09:26:13.793388 (+ 52us) client_negotiation.cc:240] Negotiation successful
0620 09:26:13.793433 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":79,"threads_started":1}
I20260620 09:26:13.793495 18898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:13.790989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.129:44149 (local address 127.16.43.130:41977)
0620 09:26:13.791152 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:13.791154 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:13.791162 (+ 8us) server_negotiation.cc:407] Connection header received
0620 09:26:13.791186 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:13.791187 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:13.791197 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:13.791281 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:26:13.791391 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:13.791980 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:13.792742 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:13.792967 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:13.793045 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:13.793227 (+ 182us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:26:13.793262 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:26:13.793332 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:13.793392 (+ 60us) server_negotiation.cc:299] Negotiation successful
0620 09:26:13.793414 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":88,"threads_started":1}
I20260620 09:26:13.795446 18586 raft_consensus.cc:1261] T 95bbd4529955431ca482f0d4bdbadc85 P 41789aa446ae4c5bbcda631441a01566 [term 1 FOLLOWER]: Refusing update from remote peer 9e86ec40a36f496fbb75462f500f4035: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:26:13.795446 18726 raft_consensus.cc:1261] T 95bbd4529955431ca482f0d4bdbadc85 P b9807e3fbedb4973a065b0c81113fa6a [term 1 FOLLOWER]: Refusing update from remote peer 9e86ec40a36f496fbb75462f500f4035: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:26:13.795702 18891 consensus_queue.cc:1035] T 95bbd4529955431ca482f0d4bdbadc85 P 9e86ec40a36f496fbb75462f500f4035 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41789aa446ae4c5bbcda631441a01566" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 41977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:26:13.795866 18887 consensus_queue.cc:1035] T 95bbd4529955431ca482f0d4bdbadc85 P 9e86ec40a36f496fbb75462f500f4035 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9807e3fbedb4973a065b0c81113fa6a" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 41849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:26:13.803279 18894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:13.800677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.130:44997 (local address 127.16.43.129:42805)
0620 09:26:13.800714 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:13.800717 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:13.800820 (+ 103us) server_negotiation.cc:407] Connection header received
0620 09:26:13.800907 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:13.800908 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:13.800924 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:13.800966 (+ 42us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:26:13.801116 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:13.801560 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:13.802418 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:13.802742 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:13.802831 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:13.802986 (+ 155us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:26:13.803019 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:26:13.803089 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:13.803207 (+ 118us) server_negotiation.cc:299] Negotiation successful
0620 09:26:13.803227 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260620 09:26:13.803354 18902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:13.800543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.129:42805 (local address 127.16.43.130:44997)
0620 09:26:13.800735 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:13.800749 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:13.800829 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:13.800975 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:13.800976 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:13.800978 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 09:26:13.801034 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:13.801036 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:13.801633 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:13.801634 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:13.802304 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:13.802307 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:13.802829 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:13.802830 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:13.802900 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:13.802906 (+ 6us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 09:26:13.803120 (+ 214us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 09:26:13.803123 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 09:26:13.803200 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 09:26:13.803261 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":118,"threads_started":1}
I20260620 09:26:13.810081 18287 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56572:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:26:13.810209 18287 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56572:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:26:13.810840 18287 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 21cc679fd523463da96d6adcab8040db: Sending DeleteTablet for 3 replicas of tablet 95bbd4529955431ca482f0d4bdbadc85
I20260620 09:26:13.811131 18706 tablet_service.cc:1514] Processing DeleteTablet for tablet 95bbd4529955431ca482f0d4bdbadc85 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:26:13 UTC) from {username='kudu', principal='kudu/127.16.43.190@KRBTEST.COM'} at 127.0.0.1:52330
I20260620 09:26:13.811136 18426 tablet_service.cc:1514] Processing DeleteTablet for tablet 95bbd4529955431ca482f0d4bdbadc85 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:26:13 UTC) from {username='kudu', principal='kudu/127.16.43.190@KRBTEST.COM'} at 127.0.0.1:53164
I20260620 09:26:13.811164 18565 tablet_service.cc:1514] Processing DeleteTablet for tablet 95bbd4529955431ca482f0d4bdbadc85 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:26:13 UTC) from {username='kudu', principal='kudu/127.16.43.190@KRBTEST.COM'} at 127.0.0.1:53952
I20260620 09:26:13.811240 18875 tablet_replica.cc:330] T 95bbd4529955431ca482f0d4bdbadc85 P 41789aa446ae4c5bbcda631441a01566: stopping tablet replica
I20260620 09:26:13.811242 18874 tablet_replica.cc:330] T 95bbd4529955431ca482f0d4bdbadc85 P b9807e3fbedb4973a065b0c81113fa6a: stopping tablet replica
I20260620 09:26:13.811298 18875 raft_consensus.cc:2229] T 95bbd4529955431ca482f0d4bdbadc85 P 41789aa446ae4c5bbcda631441a01566 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:26:13.811304 18874 raft_consensus.cc:2229] T 95bbd4529955431ca482f0d4bdbadc85 P b9807e3fbedb4973a065b0c81113fa6a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:26:13.811339 18874 raft_consensus.cc:2258] T 95bbd4529955431ca482f0d4bdbadc85 P b9807e3fbedb4973a065b0c81113fa6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:26:13.811342 18875 raft_consensus.cc:2258] T 95bbd4529955431ca482f0d4bdbadc85 P 41789aa446ae4c5bbcda631441a01566 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:26:13.811403 18873 tablet_replica.cc:330] T 95bbd4529955431ca482f0d4bdbadc85 P 9e86ec40a36f496fbb75462f500f4035: stopping tablet replica
I20260620 09:26:13.811456 18873 raft_consensus.cc:2229] T 95bbd4529955431ca482f0d4bdbadc85 P 9e86ec40a36f496fbb75462f500f4035 [term 1 LEADER]: Raft consensus shutting down.
I20260620 09:26:13.811550 18873 raft_consensus.cc:2258] T 95bbd4529955431ca482f0d4bdbadc85 P 9e86ec40a36f496fbb75462f500f4035 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:26:13.811695 18875 ts_tablet_manager.cc:1905] T 95bbd4529955431ca482f0d4bdbadc85 P 41789aa446ae4c5bbcda631441a01566: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:26:13.811944 18874 ts_tablet_manager.cc:1905] T 95bbd4529955431ca482f0d4bdbadc85 P b9807e3fbedb4973a065b0c81113fa6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:26:13.813277 18875 ts_tablet_manager.cc:1918] T 95bbd4529955431ca482f0d4bdbadc85 P 41789aa446ae4c5bbcda631441a01566: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 09:26:13.813347 18875 log.cc:1196] T 95bbd4529955431ca482f0d4bdbadc85 P 41789aa446ae4c5bbcda631441a01566: Deleting WAL directory at /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781947562949448-16558-0/minicluster-data/ts-1/wal/wals/95bbd4529955431ca482f0d4bdbadc85
I20260620 09:26:13.813427 18874 ts_tablet_manager.cc:1918] T 95bbd4529955431ca482f0d4bdbadc85 P b9807e3fbedb4973a065b0c81113fa6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 09:26:13.813485 18874 log.cc:1196] T 95bbd4529955431ca482f0d4bdbadc85 P b9807e3fbedb4973a065b0c81113fa6a: Deleting WAL directory at /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781947562949448-16558-0/minicluster-data/ts-2/wal/wals/95bbd4529955431ca482f0d4bdbadc85
I20260620 09:26:13.813572 18875 ts_tablet_manager.cc:1939] T 95bbd4529955431ca482f0d4bdbadc85 P 41789aa446ae4c5bbcda631441a01566: Deleting consensus metadata
I20260620 09:26:13.813704 18874 ts_tablet_manager.cc:1939] T 95bbd4529955431ca482f0d4bdbadc85 P b9807e3fbedb4973a065b0c81113fa6a: Deleting consensus metadata
I20260620 09:26:13.814133 18275 catalog_manager.cc:4841] TS b9807e3fbedb4973a065b0c81113fa6a (127.16.43.131:41849): tablet 95bbd4529955431ca482f0d4bdbadc85 (table test-table [id=ac69f708b390417babb362478e86cb49]) successfully deleted
I20260620 09:26:13.814230 18275 catalog_manager.cc:4841] TS 41789aa446ae4c5bbcda631441a01566 (127.16.43.130:41977): tablet 95bbd4529955431ca482f0d4bdbadc85 (table test-table [id=ac69f708b390417babb362478e86cb49]) successfully deleted
I20260620 09:26:13.814317 18873 ts_tablet_manager.cc:1905] T 95bbd4529955431ca482f0d4bdbadc85 P 9e86ec40a36f496fbb75462f500f4035: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:26:13.815644 18873 ts_tablet_manager.cc:1918] T 95bbd4529955431ca482f0d4bdbadc85 P 9e86ec40a36f496fbb75462f500f4035: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 09:26:13.815718 18873 log.cc:1196] T 95bbd4529955431ca482f0d4bdbadc85 P 9e86ec40a36f496fbb75462f500f4035: Deleting WAL directory at /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781947562949448-16558-0/minicluster-data/ts-0/wal/wals/95bbd4529955431ca482f0d4bdbadc85
I20260620 09:26:13.815932 18873 ts_tablet_manager.cc:1939] T 95bbd4529955431ca482f0d4bdbadc85 P 9e86ec40a36f496fbb75462f500f4035: Deleting consensus metadata
I20260620 09:26:13.815881 18287 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56572:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 09:26:13.816036 18287 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 09:26:13.816195 18274 catalog_manager.cc:4841] TS 9e86ec40a36f496fbb75462f500f4035 (127.16.43.129:42805): tablet 95bbd4529955431ca482f0d4bdbadc85 (table test-table [id=ac69f708b390417babb362478e86cb49]) successfully deleted
I20260620 09:26:13.817265 18426 tablet_service.cc:1467] Processing CreateTablet for tablet c45eccca6b5b44fd9ddef65e24d73608 (DEFAULT_TABLE table=test-table [id=b0fd59a2070641f7998dfef374fa2e81]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:26:13.817262 18565 tablet_service.cc:1467] Processing CreateTablet for tablet c45eccca6b5b44fd9ddef65e24d73608 (DEFAULT_TABLE table=test-table [id=b0fd59a2070641f7998dfef374fa2e81]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:26:13.817266 18706 tablet_service.cc:1467] Processing CreateTablet for tablet c45eccca6b5b44fd9ddef65e24d73608 (DEFAULT_TABLE table=test-table [id=b0fd59a2070641f7998dfef374fa2e81]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:26:13.817373 18706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c45eccca6b5b44fd9ddef65e24d73608. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:26:13.817373 18426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c45eccca6b5b44fd9ddef65e24d73608. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:26:13.817381 18565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c45eccca6b5b44fd9ddef65e24d73608. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:26:13.818544 18884 tablet_bootstrap.cc:492] T c45eccca6b5b44fd9ddef65e24d73608 P 9e86ec40a36f496fbb75462f500f4035: Bootstrap starting.
I20260620 09:26:13.818567 18886 tablet_bootstrap.cc:492] T c45eccca6b5b44fd9ddef65e24d73608 P b9807e3fbedb4973a065b0c81113fa6a: Bootstrap starting.
I20260620 09:26:13.818688 18889 tablet_bootstrap.cc:492] T c45eccca6b5b44fd9ddef65e24d73608 P 41789aa446ae4c5bbcda631441a01566: Bootstrap starting.
I20260620 09:26:13.818903 18886 tablet_bootstrap.cc:654] T c45eccca6b5b44fd9ddef65e24d73608 P b9807e3fbedb4973a065b0c81113fa6a: Neither blocks nor log segments found. Creating new log.
I20260620 09:26:13.818934 18884 tablet_bootstrap.cc:654] T c45eccca6b5b44fd9ddef65e24d73608 P 9e86ec40a36f496fbb75462f500f4035: Neither blocks nor log segments found. Creating new log.
I20260620 09:26:13.819001 18889 tablet_bootstrap.cc:654] T c45eccca6b5b44fd9ddef65e24d73608 P 41789aa446ae4c5bbcda631441a01566: Neither blocks nor log segments found. Creating new log.
I20260620 09:26:13.819670 18884 tablet_bootstrap.cc:492] T c45eccca6b5b44fd9ddef65e24d73608 P 9e86ec40a36f496fbb75462f500f4035: No bootstrap required, opened a new log
I20260620 09:26:13.819725 18884 ts_tablet_manager.cc:1397] T c45eccca6b5b44fd9ddef65e24d73608 P 9e86ec40a36f496fbb75462f500f4035: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 09:26:13.819850 18884 raft_consensus.cc:348] T c45eccca6b5b44fd9ddef65e24d73608 P 9e86ec40a36f496fbb75462f500f4035 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e86ec40a36f496fbb75462f500f4035" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 42805 } } peers { permanent_uuid: "41789aa446ae4c5bbcda631441a01566" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 41977 } } peers { permanent_uuid: "b9807e3fbedb4973a065b0c81113fa6a" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 41849 } }
I20260620 09:26:13.819901 18884 raft_consensus.cc:374] T c45eccca6b5b44fd9ddef65e24d73608 P 9e86ec40a36f496fbb75462f500f4035 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:26:13.819916 18884 raft_consensus.cc:729] T c45eccca6b5b44fd9ddef65e24d73608 P 9e86ec40a36f496fbb75462f500f4035 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e86ec40a36f496fbb75462f500f4035, State: Initialized, Role: FOLLOWER
I20260620 09:26:13.819921 18889 tablet_bootstrap.cc:492] T c45eccca6b5b44fd9ddef65e24d73608 P 41789aa446ae4c5bbcda631441a01566: No bootstrap required, opened a new log
I20260620 09:26:13.819964 18889 ts_tablet_manager.cc:1397] T c45eccca6b5b44fd9ddef65e24d73608 P 41789aa446ae4c5bbcda631441a01566: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260620 09:26:13.819977 18884 consensus_queue.cc:260] T c45eccca6b5b44fd9ddef65e24d73608 P 9e86ec40a36f496fbb75462f500f4035 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e86ec40a36f496fbb75462f500f4035" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 42805 } } peers { permanent_uuid: "41789aa446ae4c5bbcda631441a01566" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 41977 } } peers { permanent_uuid: "b9807e3fbedb4973a065b0c81113fa6a" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 41849 } }
I20260620 09:26:13.820070 18884 ts_tablet_manager.cc:1428] T c45eccca6b5b44fd9ddef65e24d73608 P 9e86ec40a36f496fbb75462f500f4035: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:13.820063 18889 raft_consensus.cc:348] T c45eccca6b5b44fd9ddef65e24d73608 P 41789aa446ae4c5bbcda631441a01566 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e86ec40a36f496fbb75462f500f4035" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 42805 } } peers { permanent_uuid: "41789aa446ae4c5bbcda631441a01566" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 41977 } } peers { permanent_uuid: "b9807e3fbedb4973a065b0c81113fa6a" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 41849 } }
I20260620 09:26:13.820123 18889 raft_consensus.cc:374] T c45eccca6b5b44fd9ddef65e24d73608 P 41789aa446ae4c5bbcda631441a01566 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:26:13.820143 18889 raft_consensus.cc:729] T c45eccca6b5b44fd9ddef65e24d73608 P 41789aa446ae4c5bbcda631441a01566 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41789aa446ae4c5bbcda631441a01566, State: Initialized, Role: FOLLOWER
I20260620 09:26:13.820190 18889 consensus_queue.cc:260] T c45eccca6b5b44fd9ddef65e24d73608 P 41789aa446ae4c5bbcda631441a01566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e86ec40a36f496fbb75462f500f4035" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 42805 } } peers { permanent_uuid: "41789aa446ae4c5bbcda631441a01566" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 41977 } } peers { permanent_uuid: "b9807e3fbedb4973a065b0c81113fa6a" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 41849 } }
I20260620 09:26:13.820282 18889 ts_tablet_manager.cc:1428] T c45eccca6b5b44fd9ddef65e24d73608 P 41789aa446ae4c5bbcda631441a01566: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:13.820549 18886 tablet_bootstrap.cc:492] T c45eccca6b5b44fd9ddef65e24d73608 P b9807e3fbedb4973a065b0c81113fa6a: No bootstrap required, opened a new log
I20260620 09:26:13.820608 18886 ts_tablet_manager.cc:1397] T c45eccca6b5b44fd9ddef65e24d73608 P b9807e3fbedb4973a065b0c81113fa6a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 09:26:13.820734 18886 raft_consensus.cc:348] T c45eccca6b5b44fd9ddef65e24d73608 P b9807e3fbedb4973a065b0c81113fa6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e86ec40a36f496fbb75462f500f4035" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 42805 } } peers { permanent_uuid: "41789aa446ae4c5bbcda631441a01566" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 41977 } } peers { permanent_uuid: "b9807e3fbedb4973a065b0c81113fa6a" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 41849 } }
I20260620 09:26:13.820806 18886 raft_consensus.cc:374] T c45eccca6b5b44fd9ddef65e24d73608 P b9807e3fbedb4973a065b0c81113fa6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:26:13.820829 18886 raft_consensus.cc:729] T c45eccca6b5b44fd9ddef65e24d73608 P b9807e3fbedb4973a065b0c81113fa6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9807e3fbedb4973a065b0c81113fa6a, State: Initialized, Role: FOLLOWER
I20260620 09:26:13.820879 18886 consensus_queue.cc:260] T c45eccca6b5b44fd9ddef65e24d73608 P b9807e3fbedb4973a065b0c81113fa6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e86ec40a36f496fbb75462f500f4035" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 42805 } } peers { permanent_uuid: "41789aa446ae4c5bbcda631441a01566" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 41977 } } peers { permanent_uuid: "b9807e3fbedb4973a065b0c81113fa6a" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 41849 } }
I20260620 09:26:13.820971 18886 ts_tablet_manager.cc:1428] T c45eccca6b5b44fd9ddef65e24d73608 P b9807e3fbedb4973a065b0c81113fa6a: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 09:26:14.016451 18850 raft_consensus.cc:482] T c45eccca6b5b44fd9ddef65e24d73608 P b9807e3fbedb4973a065b0c81113fa6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:26:14.016561 18850 raft_consensus.cc:504] T c45eccca6b5b44fd9ddef65e24d73608 P b9807e3fbedb4973a065b0c81113fa6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e86ec40a36f496fbb75462f500f4035" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 42805 } } peers { permanent_uuid: "41789aa446ae4c5bbcda631441a01566" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 41977 } } peers { permanent_uuid: "b9807e3fbedb4973a065b0c81113fa6a" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 41849 } }
I20260620 09:26:14.016728 18850 leader_election.cc:288] T c45eccca6b5b44fd9ddef65e24d73608 P b9807e3fbedb4973a065b0c81113fa6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9e86ec40a36f496fbb75462f500f4035 (127.16.43.129:42805), 41789aa446ae4c5bbcda631441a01566 (127.16.43.130:41977)
I20260620 09:26:14.016975 18446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c45eccca6b5b44fd9ddef65e24d73608" candidate_uuid: "b9807e3fbedb4973a065b0c81113fa6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e86ec40a36f496fbb75462f500f4035" is_pre_election: true
I20260620 09:26:14.017115 18446 raft_consensus.cc:2454] T c45eccca6b5b44fd9ddef65e24d73608 P 9e86ec40a36f496fbb75462f500f4035 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9807e3fbedb4973a065b0c81113fa6a in term 0.
I20260620 09:26:14.017024 18586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c45eccca6b5b44fd9ddef65e24d73608" candidate_uuid: "b9807e3fbedb4973a065b0c81113fa6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41789aa446ae4c5bbcda631441a01566" is_pre_election: true
I20260620 09:26:14.017177 18586 raft_consensus.cc:2454] T c45eccca6b5b44fd9ddef65e24d73608 P 41789aa446ae4c5bbcda631441a01566 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9807e3fbedb4973a065b0c81113fa6a in term 0.
I20260620 09:26:14.017315 18658 leader_election.cc:302] T c45eccca6b5b44fd9ddef65e24d73608 P b9807e3fbedb4973a065b0c81113fa6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9e86ec40a36f496fbb75462f500f4035, b9807e3fbedb4973a065b0c81113fa6a; no voters:
I20260620 09:26:14.017437 18850 raft_consensus.cc:2781] T c45eccca6b5b44fd9ddef65e24d73608 P b9807e3fbedb4973a065b0c81113fa6a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 09:26:14.017484 18850 raft_consensus.cc:482] T c45eccca6b5b44fd9ddef65e24d73608 P b9807e3fbedb4973a065b0c81113fa6a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 09:26:14.017525 18850 raft_consensus.cc:3037] T c45eccca6b5b44fd9ddef65e24d73608 P b9807e3fbedb4973a065b0c81113fa6a [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:26:14.018311 18850 raft_consensus.cc:504] T c45eccca6b5b44fd9ddef65e24d73608 P b9807e3fbedb4973a065b0c81113fa6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e86ec40a36f496fbb75462f500f4035" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 42805 } } peers { permanent_uuid: "41789aa446ae4c5bbcda631441a01566" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 41977 } } peers { permanent_uuid: "b9807e3fbedb4973a065b0c81113fa6a" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 41849 } }
I20260620 09:26:14.018453 18850 leader_election.cc:288] T c45eccca6b5b44fd9ddef65e24d73608 P b9807e3fbedb4973a065b0c81113fa6a [CANDIDATE]: Term 1 election: Requested vote from peers 9e86ec40a36f496fbb75462f500f4035 (127.16.43.129:42805), 41789aa446ae4c5bbcda631441a01566 (127.16.43.130:41977)
I20260620 09:26:14.018635 18446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c45eccca6b5b44fd9ddef65e24d73608" candidate_uuid: "b9807e3fbedb4973a065b0c81113fa6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e86ec40a36f496fbb75462f500f4035"
I20260620 09:26:14.018682 18586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c45eccca6b5b44fd9ddef65e24d73608" candidate_uuid: "b9807e3fbedb4973a065b0c81113fa6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41789aa446ae4c5bbcda631441a01566"
I20260620 09:26:14.018750 18446 raft_consensus.cc:3037] T c45eccca6b5b44fd9ddef65e24d73608 P 9e86ec40a36f496fbb75462f500f4035 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:26:14.018755 18586 raft_consensus.cc:3037] T c45eccca6b5b44fd9ddef65e24d73608 P 41789aa446ae4c5bbcda631441a01566 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:26:14.019634 18586 raft_consensus.cc:2454] T c45eccca6b5b44fd9ddef65e24d73608 P 41789aa446ae4c5bbcda631441a01566 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9807e3fbedb4973a065b0c81113fa6a in term 1.
I20260620 09:26:14.019634 18446 raft_consensus.cc:2454] T c45eccca6b5b44fd9ddef65e24d73608 P 9e86ec40a36f496fbb75462f500f4035 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9807e3fbedb4973a065b0c81113fa6a in term 1.
I20260620 09:26:14.019821 18658 leader_election.cc:302] T c45eccca6b5b44fd9ddef65e24d73608 P b9807e3fbedb4973a065b0c81113fa6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9e86ec40a36f496fbb75462f500f4035, b9807e3fbedb4973a065b0c81113fa6a; no voters:
I20260620 09:26:14.019941 18850 raft_consensus.cc:2781] T c45eccca6b5b44fd9ddef65e24d73608 P b9807e3fbedb4973a065b0c81113fa6a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:26:14.020015 18850 raft_consensus.cc:686] T c45eccca6b5b44fd9ddef65e24d73608 P b9807e3fbedb4973a065b0c81113fa6a [term 1 LEADER]: Becoming Leader. State: Replica: b9807e3fbedb4973a065b0c81113fa6a, State: Running, Role: LEADER
I20260620 09:26:14.020066 18850 consensus_queue.cc:237] T c45eccca6b5b44fd9ddef65e24d73608 P b9807e3fbedb4973a065b0c81113fa6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e86ec40a36f496fbb75462f500f4035" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 42805 } } peers { permanent_uuid: "41789aa446ae4c5bbcda631441a01566" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 41977 } } peers { permanent_uuid: "b9807e3fbedb4973a065b0c81113fa6a" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 41849 } }
I20260620 09:26:14.020758 18287 catalog_manager.cc:5495] T c45eccca6b5b44fd9ddef65e24d73608 P b9807e3fbedb4973a065b0c81113fa6a reported cstate change: term changed from 0 to 1, leader changed from <none> to b9807e3fbedb4973a065b0c81113fa6a (127.16.43.131). New cstate: current_term: 1 leader_uuid: "b9807e3fbedb4973a065b0c81113fa6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e86ec40a36f496fbb75462f500f4035" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 42805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "41789aa446ae4c5bbcda631441a01566" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 41977 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9807e3fbedb4973a065b0c81113fa6a" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 41849 } health_report { overall_health: HEALTHY } } }
I20260620 09:26:14.051029 18909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:14.048905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52352 (local address 127.16.43.131:41849)
0620 09:26:14.049076 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:14.049078 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:14.049086 (+ 8us) server_negotiation.cc:407] Connection header received
0620 09:26:14.049119 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:14.049120 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:14.049130 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:14.049183 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:26:14.049280 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:14.049749 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:14.050432 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:14.050615 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:14.050666 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:14.050769 (+ 103us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:26:14.050793 (+ 24us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:26:14.050821 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:14.050859 (+ 38us) server_negotiation.cc:299] Negotiation successful
0620 09:26:14.050874 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":116,"threads_started":1}
I20260620 09:26:14.053550 18446 raft_consensus.cc:1261] T c45eccca6b5b44fd9ddef65e24d73608 P 9e86ec40a36f496fbb75462f500f4035 [term 1 FOLLOWER]: Refusing update from remote peer b9807e3fbedb4973a065b0c81113fa6a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:26:14.053570 18586 raft_consensus.cc:1261] T c45eccca6b5b44fd9ddef65e24d73608 P 41789aa446ae4c5bbcda631441a01566 [term 1 FOLLOWER]: Refusing update from remote peer b9807e3fbedb4973a065b0c81113fa6a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:26:14.053802 18850 consensus_queue.cc:1035] T c45eccca6b5b44fd9ddef65e24d73608 P b9807e3fbedb4973a065b0c81113fa6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "41789aa446ae4c5bbcda631441a01566" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 41977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:26:14.053951 18850 consensus_queue.cc:1035] T c45eccca6b5b44fd9ddef65e24d73608 P b9807e3fbedb4973a065b0c81113fa6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e86ec40a36f496fbb75462f500f4035" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 42805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 09:26:14.057996 18856 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 09:26:15.578117 18288 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56572:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:26:15.578233 18288 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56572:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:26:15.579085 18288 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 21cc679fd523463da96d6adcab8040db: Sending DeleteTablet for 3 replicas of tablet c45eccca6b5b44fd9ddef65e24d73608
I20260620 09:26:15.579363 18565 tablet_service.cc:1514] Processing DeleteTablet for tablet c45eccca6b5b44fd9ddef65e24d73608 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:26:15 UTC) from {username='kudu', principal='kudu/127.16.43.190@KRBTEST.COM'} at 127.0.0.1:53952
I20260620 09:26:15.579361 18426 tablet_service.cc:1514] Processing DeleteTablet for tablet c45eccca6b5b44fd9ddef65e24d73608 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:26:15 UTC) from {username='kudu', principal='kudu/127.16.43.190@KRBTEST.COM'} at 127.0.0.1:53164
I20260620 09:26:15.579586 18934 tablet_replica.cc:330] T c45eccca6b5b44fd9ddef65e24d73608 P 9e86ec40a36f496fbb75462f500f4035: stopping tablet replica
I20260620 09:26:15.579609 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 18361
I20260620 09:26:15.579619 18706 tablet_service.cc:1514] Processing DeleteTablet for tablet c45eccca6b5b44fd9ddef65e24d73608 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:26:15 UTC) from {username='kudu', principal='kudu/127.16.43.190@KRBTEST.COM'} at 127.0.0.1:52330
I20260620 09:26:15.580250 18935 tablet_replica.cc:330] T c45eccca6b5b44fd9ddef65e24d73608 P 41789aa446ae4c5bbcda631441a01566: stopping tablet replica
I20260620 09:26:15.580323 18935 raft_consensus.cc:2229] T c45eccca6b5b44fd9ddef65e24d73608 P 41789aa446ae4c5bbcda631441a01566 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:26:15.580355 18936 tablet_replica.cc:330] T c45eccca6b5b44fd9ddef65e24d73608 P b9807e3fbedb4973a065b0c81113fa6a: stopping tablet replica
I20260620 09:26:15.580374 18935 raft_consensus.cc:2258] T c45eccca6b5b44fd9ddef65e24d73608 P 41789aa446ae4c5bbcda631441a01566 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:26:15.580420 18936 raft_consensus.cc:2229] T c45eccca6b5b44fd9ddef65e24d73608 P b9807e3fbedb4973a065b0c81113fa6a [term 1 LEADER]: Raft consensus shutting down.
I20260620 09:26:15.580487 18936 raft_consensus.cc:2258] T c45eccca6b5b44fd9ddef65e24d73608 P b9807e3fbedb4973a065b0c81113fa6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:26:15.580691 18935 ts_tablet_manager.cc:1905] T c45eccca6b5b44fd9ddef65e24d73608 P 41789aa446ae4c5bbcda631441a01566: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:26:15.580725 18936 ts_tablet_manager.cc:1905] T c45eccca6b5b44fd9ddef65e24d73608 P b9807e3fbedb4973a065b0c81113fa6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:26:15.582242 18936 ts_tablet_manager.cc:1918] T c45eccca6b5b44fd9ddef65e24d73608 P b9807e3fbedb4973a065b0c81113fa6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 09:26:15.582320 18936 log.cc:1196] T c45eccca6b5b44fd9ddef65e24d73608 P b9807e3fbedb4973a065b0c81113fa6a: Deleting WAL directory at /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781947562949448-16558-0/minicluster-data/ts-2/wal/wals/c45eccca6b5b44fd9ddef65e24d73608
I20260620 09:26:15.582559 18936 ts_tablet_manager.cc:1939] T c45eccca6b5b44fd9ddef65e24d73608 P b9807e3fbedb4973a065b0c81113fa6a: Deleting consensus metadata
I20260620 09:26:15.582680 18935 ts_tablet_manager.cc:1918] T c45eccca6b5b44fd9ddef65e24d73608 P 41789aa446ae4c5bbcda631441a01566: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 09:26:15.582746 18935 log.cc:1196] T c45eccca6b5b44fd9ddef65e24d73608 P 41789aa446ae4c5bbcda631441a01566: Deleting WAL directory at /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781947562949448-16558-0/minicluster-data/ts-1/wal/wals/c45eccca6b5b44fd9ddef65e24d73608
I20260620 09:26:15.582976 18935 ts_tablet_manager.cc:1939] T c45eccca6b5b44fd9ddef65e24d73608 P 41789aa446ae4c5bbcda631441a01566: Deleting consensus metadata
I20260620 09:26:15.583204 18275 catalog_manager.cc:4841] TS b9807e3fbedb4973a065b0c81113fa6a (127.16.43.131:41849): tablet c45eccca6b5b44fd9ddef65e24d73608 (table test-table [id=b0fd59a2070641f7998dfef374fa2e81]) successfully deleted
I20260620 09:26:15.583424 18275 catalog_manager.cc:4841] TS 41789aa446ae4c5bbcda631441a01566 (127.16.43.130:41977): tablet c45eccca6b5b44fd9ddef65e24d73608 (table test-table [id=b0fd59a2070641f7998dfef374fa2e81]) successfully deleted
W20260620 09:26:15.587674 18274 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.43.129:42805 (error 108)
I20260620 09:26:15.588270 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 18501
I20260620 09:26:15.589895 18938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:15.588112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.129:42805 (local address 127.0.0.1:53176)
0620 09:26:15.588258 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:15.588306 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.129:42805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":96,"threads_started":1}
W20260620 09:26:15.590081 18274 catalog_manager.cc:4584] TS 9e86ec40a36f496fbb75462f500f4035 (127.16.43.129:42805): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c45eccca6b5b44fd9ddef65e24d73608: Network error: Client connection negotiation failed: client connection to 127.16.43.129:42805: connect: Connection refused (error 111)
I20260620 09:26:15.595229 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 18641
I20260620 09:26:15.600935 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 18257
2026-06-20T09:26:15Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5635 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781947562949448-16558-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:26:15 dist-test-slave-706q krb5kdc[18941](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:26:15 dist-test-slave-706q krb5kdc[18941](info): set up 2 sockets
Jun 20 09:26:15 dist-test-slave-706q krb5kdc[18941](info): commencing operation
krb5kdc: starting...
W20260620 09:26:16.652449 16558 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781947562949448-16558-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781947562949448-16558-0/krb5kdc/test-user.keytab.
Jun 20 09:26:16 dist-test-slave-706q krb5kdc[18941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947576, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:26:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:26:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.43.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.43.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.43.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.190.keytab.
Entry for principal kudu/127.16.43.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.190.keytab.
I20260620 09:26:16.780140 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781947562949448-16558-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781947562949448-16558-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781947562949448-16558-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.43.190:33283
--webserver_interface=127.16.43.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.190.keytab
--principal=kudu/127.16.43.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.43.148:41215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.43.190:33283
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781947562949448-16558-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781947562949448-16558-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781947562949448-16558-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:26:16.862270 18954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:26:16.862479 18954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:26:16.862502 18954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:26:16.863802 18954 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:26:16.863850 18954 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:26:16.863865 18954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:26:16.863878 18954 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:26:16.863890 18954 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:26:16.865527 18954 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:41215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781947562949448-16558-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781947562949448-16558-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.43.190:33283
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.190.keytab
--principal=kudu/127.16.43.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.43.190:33283
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.43.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18954
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781947562949448-16558-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:26:16.865928 18954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:26:16.866225 18954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:26:16.871464 18960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:26:16.871578 18959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:26:16.871476 18962 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:26:16.871515 18954 server_base.cc:1034] running on GCE node
I20260620 09:26:16.871879 18954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:26:16.872201 18954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:26:16.873376 18954 hybrid_clock.cc:648] HybridClock initialized: now 1781947576873353 us; error 40 us; skew 500 ppm
Jun 20 09:26:16 dist-test-slave-706q krb5kdc[18941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947576, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:26:16.881485 18954 init.cc:377] Logged in from keytab as kudu/127.16.43.190@KRBTEST.COM (short username kudu)
I20260620 09:26:16.882124 18954 webserver.cc:460] Webserver started at http://127.16.43.190:34517/ using document root <none> and password file <none>
I20260620 09:26:16.882401 18954 fs_manager.cc:359] Metadata directory not provided
I20260620 09:26:16.882483 18954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:26:16.882642 18954 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:26:16.883872 18954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781947562949448-16558-0/minicluster-data/master-0/data/instance:
uuid: "9937d641a4f345f9842fbcca284fbd03"
format_stamp: "Formatted at 2026-06-20 09:26:16 on dist-test-slave-706q"
server_key: "9d8d0ff233d1fd2059805b3b41f3d29d"
server_key_iv: "e05f18e28ae658510114adee015d49e2"
server_key_version: "encryptionkey@0"
I20260620 09:26:16.884243 18954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781947562949448-16558-0/minicluster-data/master-0/wal/instance:
uuid: "9937d641a4f345f9842fbcca284fbd03"
format_stamp: "Formatted at 2026-06-20 09:26:16 on dist-test-slave-706q"
server_key: "9d8d0ff233d1fd2059805b3b41f3d29d"
server_key_iv: "e05f18e28ae658510114adee015d49e2"
server_key_version: "encryptionkey@0"
I20260620 09:26:16.885751 18954 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 09:26:16.886509 18969 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:16.886687 18954 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 09:26:16.886813 18954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781947562949448-16558-0/minicluster-data/master-0/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781947562949448-16558-0/minicluster-data/master-0/wal
uuid: "9937d641a4f345f9842fbcca284fbd03"
format_stamp: "Formatted at 2026-06-20 09:26:16 on dist-test-slave-706q"
server_key: "9d8d0ff233d1fd2059805b3b41f3d29d"
server_key_iv: "e05f18e28ae658510114adee015d49e2"
server_key_version: "encryptionkey@0"
I20260620 09:26:16.886893 18954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781947562949448-16558-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781947562949448-16558-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781947562949448-16558-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:26:16.900970 18954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:26:16.903229 18954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:26:16.903437 18954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:26:16.907768 18954 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.190:33283
I20260620 09:26:16.907797 19021 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.190:33283 every 8 connection(s)
I20260620 09:26:16.908162 18954 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
I20260620 09:26:16.909032 19022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:26:16.911351 19022 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9937d641a4f345f9842fbcca284fbd03: Bootstrap starting.
I20260620 09:26:16.912039 19022 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9937d641a4f345f9842fbcca284fbd03: Neither blocks nor log segments found. Creating new log.
I20260620 09:26:16.912292 19022 log.cc:824] T 00000000000000000000000000000000 P 9937d641a4f345f9842fbcca284fbd03: Log is configured to *not* fsync() on all Append() calls
I20260620 09:26:16.913010 19022 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9937d641a4f345f9842fbcca284fbd03: No bootstrap required, opened a new log
I20260620 09:26:16.914577 19022 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9937d641a4f345f9842fbcca284fbd03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9937d641a4f345f9842fbcca284fbd03" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 33283 } }
I20260620 09:26:16.914762 19022 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9937d641a4f345f9842fbcca284fbd03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:26:16.914788 19022 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9937d641a4f345f9842fbcca284fbd03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9937d641a4f345f9842fbcca284fbd03, State: Initialized, Role: FOLLOWER
I20260620 09:26:16.914944 19022 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9937d641a4f345f9842fbcca284fbd03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9937d641a4f345f9842fbcca284fbd03" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 33283 } }
I20260620 09:26:16.915031 19022 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9937d641a4f345f9842fbcca284fbd03 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:26:16.915055 19022 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9937d641a4f345f9842fbcca284fbd03 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:26:16.915086 19022 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9937d641a4f345f9842fbcca284fbd03 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:26:16.915436 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 18954
I20260620 09:26:16.915581 16558 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781947562949448-16558-0/minicluster-data/master-0/wal/instance
I20260620 09:26:16.915719 19022 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9937d641a4f345f9842fbcca284fbd03 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9937d641a4f345f9842fbcca284fbd03" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 33283 } }
I20260620 09:26:16.915824 16558 external_mini_cluster.cc:1428] Setting key b7a725d819fbd70a73aa71116bd9f8b7
I20260620 09:26:16.915838 19022 leader_election.cc:302] T 00000000000000000000000000000000 P 9937d641a4f345f9842fbcca284fbd03 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9937d641a4f345f9842fbcca284fbd03; no voters:
I20260620 09:26:16.916049 19022 leader_election.cc:288] T 00000000000000000000000000000000 P 9937d641a4f345f9842fbcca284fbd03 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:26:16.916074 19025 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9937d641a4f345f9842fbcca284fbd03 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:26:16.916263 19025 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9937d641a4f345f9842fbcca284fbd03 [term 1 LEADER]: Becoming Leader. State: Replica: 9937d641a4f345f9842fbcca284fbd03, State: Running, Role: LEADER
I20260620 09:26:16.916337 19022 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9937d641a4f345f9842fbcca284fbd03 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:26:16.916405 19025 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9937d641a4f345f9842fbcca284fbd03 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9937d641a4f345f9842fbcca284fbd03" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 33283 } }
I20260620 09:26:16.916749 19026 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9937d641a4f345f9842fbcca284fbd03 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9937d641a4f345f9842fbcca284fbd03" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9937d641a4f345f9842fbcca284fbd03" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 33283 } } }
I20260620 09:26:16.916816 19026 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9937d641a4f345f9842fbcca284fbd03 [sys.catalog]: This master's current role is: LEADER
I20260620 09:26:16.916728 19025 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9937d641a4f345f9842fbcca284fbd03 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9937d641a4f345f9842fbcca284fbd03. Latest consensus state: current_term: 1 leader_uuid: "9937d641a4f345f9842fbcca284fbd03" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9937d641a4f345f9842fbcca284fbd03" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 33283 } } }
I20260620 09:26:16.916942 19025 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9937d641a4f345f9842fbcca284fbd03 [sys.catalog]: This master's current role is: LEADER
I20260620 09:26:16.917526 19031 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:26:16.918123 19031 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:26:16.919900 19031 catalog_manager.cc:1347] Generated new cluster ID: 05ece502b9b84e4285d40d6adb040ea5
I20260620 09:26:16.919963 19031 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 09:26:16 dist-test-slave-706q krb5kdc[18941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947576, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.43.190@KRBTEST.COM
I20260620 09:26:16.926698 19031 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:26:16.927389 19031 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:26:16.928241 19029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:16.916517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41580 (local address 127.16.43.190:33283)
0620 09:26:16.916690 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:16.916700 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:16.916739 (+ 39us) server_negotiation.cc:407] Connection header received
0620 09:26:16.917115 (+ 376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:16.917121 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:16.917169 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:16.917367 (+ 198us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:16.918314 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:16.919839 (+ 1525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:16.920640 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:16.921247 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:16.921353 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:16.924136 (+ 2783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:16.924141 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:16.924141 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:26:16.924222 (+ 81us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:16.926303 (+ 2081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:16.926770 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:16.926774 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:16.926774 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:16.926904 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:16.927128 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:16.927129 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:16.927129 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:16.927444 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:16.927639 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:16.927800 (+ 161us) server_negotiation.cc:299] Negotiation successful
0620 09:26:16.927971 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":70,"threads_started":1}
I20260620 09:26:16.931962 19031 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9937d641a4f345f9842fbcca284fbd03: Generated new TSK 0
I20260620 09:26:16.932228 19031 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.43.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.43.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.43.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.129.keytab.
Entry for principal kudu/127.16.43.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.129.keytab.
I20260620 09:26:16.963459 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781947562949448-16558-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781947562949448-16558-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781947562949448-16558-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781947562949448-16558-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.43.129:0
--local_ip_for_outbound_sockets=127.16.43.129
--webserver_interface=127.16.43.129
--webserver_port=0
--tserver_master_addrs=127.16.43.190:33283
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.129.keytab
--principal=kudu/127.16.43.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.43.148:41215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781947562949448-16558-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781947562949448-16558-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781947562949448-16558-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:26:17.047420 19048 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:26:17.047725 19048 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:26:17.047780 19048 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:26:17.049064 19048 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:26:17.049134 19048 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:26:17.049178 19048 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.43.129
I20260620 09:26:17.050642 19048 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:41215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781947562949448-16558-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781947562949448-16558-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.129.keytab
--principal=kudu/127.16.43.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.43.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781947562949448-16558-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.43.129
--webserver_port=0
--tserver_master_addrs=127.16.43.190:33283
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19048
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.43.129
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781947562949448-16558-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:26:17.050917 19048 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:26:17.051159 19048 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:26:17.056318 19053 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:26:17.056314 19056 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:26:17.056391 19054 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:26:17.056460 19048 server_base.cc:1034] running on GCE node
I20260620 09:26:17.056659 19048 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:26:17.056898 19048 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:26:17.058056 19048 hybrid_clock.cc:648] HybridClock initialized: now 1781947577058057 us; error 47 us; skew 500 ppm
Jun 20 09:26:17 dist-test-slave-706q krb5kdc[18941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947577, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:26:17.060669 19048 init.cc:377] Logged in from keytab as kudu/127.16.43.129@KRBTEST.COM (short username kudu)
I20260620 09:26:17.061123 19048 webserver.cc:460] Webserver started at http://127.16.43.129:35087/ using document root <none> and password file <none>
I20260620 09:26:17.061324 19048 fs_manager.cc:359] Metadata directory not provided
I20260620 09:26:17.061364 19048 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:26:17.061453 19048 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:26:17.062443 19048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781947562949448-16558-0/minicluster-data/ts-0/data/instance:
uuid: "df6023e6f61d4b0f8b5189a57c054428"
format_stamp: "Formatted at 2026-06-20 09:26:17 on dist-test-slave-706q"
server_key: "0ad83a207e9f766b4638effb91226b6e"
server_key_iv: "ef4569338836b4d26b7ec13f40289a8e"
server_key_version: "encryptionkey@0"
I20260620 09:26:17.062728 19048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781947562949448-16558-0/minicluster-data/ts-0/wal/instance:
uuid: "df6023e6f61d4b0f8b5189a57c054428"
format_stamp: "Formatted at 2026-06-20 09:26:17 on dist-test-slave-706q"
server_key: "0ad83a207e9f766b4638effb91226b6e"
server_key_iv: "ef4569338836b4d26b7ec13f40289a8e"
server_key_version: "encryptionkey@0"
I20260620 09:26:17.064076 19048 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 09:26:17.064752 19063 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:17.064950 19048 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 09:26:17.065032 19048 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781947562949448-16558-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781947562949448-16558-0/minicluster-data/ts-0/wal
uuid: "df6023e6f61d4b0f8b5189a57c054428"
format_stamp: "Formatted at 2026-06-20 09:26:17 on dist-test-slave-706q"
server_key: "0ad83a207e9f766b4638effb91226b6e"
server_key_iv: "ef4569338836b4d26b7ec13f40289a8e"
server_key_version: "encryptionkey@0"
I20260620 09:26:17.065086 19048 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781947562949448-16558-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781947562949448-16558-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781947562949448-16558-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:26:17.097646 19048 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:26:17.100180 19048 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:26:17.100310 19048 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:26:17.100503 19048 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:26:17.100833 19048 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:26:17.100869 19048 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:17.100894 19048 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:26:17.100941 19048 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:17.107390 19048 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.129:38837
I20260620 09:26:17.107585 19176 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.129:38837 every 8 connection(s)
I20260620 09:26:17.107846 19048 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781947562949448-16558-0/minicluster-data/ts-0/data/info.pb
I20260620 09:26:17.108919 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 19048
I20260620 09:26:17.109014 16558 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781947562949448-16558-0/minicluster-data/ts-0/wal/instance
I20260620 09:26:17.109184 16558 external_mini_cluster.cc:1428] Setting key 20f2100a54b55c416c12c5d1bb084144
Jun 20 09:26:17 dist-test-slave-706q krb5kdc[18941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947577, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.129@KRBTEST.COM for kudu/127.16.43.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.43.130@KRBTEST.COM; defaulting to no policy
I20260620 09:26:17.118582 19029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:17.108641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.129:44583 (local address 127.16.43.190:33283)
0620 09:26:17.108689 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:17.108691 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:17.108941 (+ 250us) server_negotiation.cc:407] Connection header received
0620 09:26:17.109614 (+ 673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:17.109615 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:17.109627 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:17.109672 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:17.110975 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:17.111797 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:17.112838 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:17.113057 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:17.113149 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:17.115387 (+ 2238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:17.115388 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:17.115389 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:26:17.115403 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:17.116860 (+ 1457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:17.117410 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:17.117410 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:17.117411 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:17.117463 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:17.117793 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:17.117794 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:17.117795 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:17.117920 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:17.118033 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:17.118424 (+ 391us) server_negotiation.cc:299] Negotiation successful
0620 09:26:17.118512 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 09:26:17.118752 19179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:17.108600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:33283 (local address 127.16.43.129:44583)
0620 09:26:17.108746 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:17.108830 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:17.109497 (+ 667us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:17.109744 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:17.109748 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:17.110575 (+ 827us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:17.110867 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:17.110874 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:17.111900 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:17.111901 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:17.112714 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:17.112716 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:17.113162 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:17.113163 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:17.113309 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:17.113938 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:26:17.113946 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:26:17.115280 (+ 1334us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:26:17.116985 (+ 1705us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:17.116993 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:17.116995 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:17.117335 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:17.117535 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:17.117536 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:17.117537 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:17.117688 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:17.118031 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:26:17.118038 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:26:17.118221 (+ 183us) client_negotiation.cc:769] Sending connection context
0620 09:26:17.118368 (+ 147us) client_negotiation.cc:240] Negotiation successful
0620 09:26:17.118534 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":66,"threads_started":1}
I20260620 09:26:17.119282 19177 heartbeater.cc:347] Connected to a master server at 127.16.43.190:33283
I20260620 09:26:17.119390 19177 heartbeater.cc:464] Registering TS with master...
I20260620 09:26:17.119601 19177 heartbeater.cc:511] Master 127.16.43.190:33283 requested a full tablet report, sending...
I20260620 09:26:17.120142 18986 ts_manager.cc:194] Registered new tserver with Master: df6023e6f61d4b0f8b5189a57c054428 (127.16.43.129:38837)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.43.130@KRBTEST.COM" created.
I20260620 09:26:17.120905 18986 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.43.129@KRBTEST.COM'} at 127.16.43.129:44583
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.43.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.130.keytab.
Entry for principal kudu/127.16.43.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.130.keytab.
I20260620 09:26:17.136612 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781947562949448-16558-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781947562949448-16558-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781947562949448-16558-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781947562949448-16558-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.43.130:0
--local_ip_for_outbound_sockets=127.16.43.130
--webserver_interface=127.16.43.130
--webserver_port=0
--tserver_master_addrs=127.16.43.190:33283
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.130.keytab
--principal=kudu/127.16.43.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.43.148:41215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781947562949448-16558-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781947562949448-16558-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781947562949448-16558-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:26:17.219373 19182 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:26:17.219659 19182 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:26:17.219689 19182 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:26:17.220870 19182 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:26:17.220916 19182 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:26:17.220942 19182 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.43.130
I20260620 09:26:17.222469 19182 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:41215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781947562949448-16558-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781947562949448-16558-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.130.keytab
--principal=kudu/127.16.43.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.43.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781947562949448-16558-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.43.130
--webserver_port=0
--tserver_master_addrs=127.16.43.190:33283
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19182
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.43.130
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781947562949448-16558-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:26:17.222836 19182 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:26:17.223034 19182 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:26:17.228056 19188 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:26:17.228089 19190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:26:17.228091 19187 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:26:17.228416 19182 server_base.cc:1034] running on GCE node
I20260620 09:26:17.228641 19182 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:26:17.228840 19182 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:26:17.230014 19182 hybrid_clock.cc:648] HybridClock initialized: now 1781947577230000 us; error 28 us; skew 500 ppm
Jun 20 09:26:17 dist-test-slave-706q krb5kdc[18941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947577, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:26:17.232877 19182 init.cc:377] Logged in from keytab as kudu/127.16.43.130@KRBTEST.COM (short username kudu)
I20260620 09:26:17.233388 19182 webserver.cc:460] Webserver started at http://127.16.43.130:35667/ using document root <none> and password file <none>
I20260620 09:26:17.233644 19182 fs_manager.cc:359] Metadata directory not provided
I20260620 09:26:17.233714 19182 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:26:17.233867 19182 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:26:17.234975 19182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781947562949448-16558-0/minicluster-data/ts-1/data/instance:
uuid: "0a7a0176b3ab461fa2cc166e2d7cb6ee"
format_stamp: "Formatted at 2026-06-20 09:26:17 on dist-test-slave-706q"
server_key: "3c4bd5581ac64a4ef4c1495e32e74cce"
server_key_iv: "9d97ead34e2658bcf00c7fedffa472f8"
server_key_version: "encryptionkey@0"
I20260620 09:26:17.235399 19182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781947562949448-16558-0/minicluster-data/ts-1/wal/instance:
uuid: "0a7a0176b3ab461fa2cc166e2d7cb6ee"
format_stamp: "Formatted at 2026-06-20 09:26:17 on dist-test-slave-706q"
server_key: "3c4bd5581ac64a4ef4c1495e32e74cce"
server_key_iv: "9d97ead34e2658bcf00c7fedffa472f8"
server_key_version: "encryptionkey@0"
I20260620 09:26:17.236761 19182 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 09:26:17.237527 19197 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:17.237728 19182 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 09:26:17.237828 19182 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781947562949448-16558-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781947562949448-16558-0/minicluster-data/ts-1/wal
uuid: "0a7a0176b3ab461fa2cc166e2d7cb6ee"
format_stamp: "Formatted at 2026-06-20 09:26:17 on dist-test-slave-706q"
server_key: "3c4bd5581ac64a4ef4c1495e32e74cce"
server_key_iv: "9d97ead34e2658bcf00c7fedffa472f8"
server_key_version: "encryptionkey@0"
I20260620 09:26:17.237912 19182 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781947562949448-16558-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781947562949448-16558-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781947562949448-16558-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:26:17.254703 19182 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:26:17.257225 19182 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:26:17.257406 19182 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:26:17.257651 19182 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:26:17.257978 19182 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:26:17.258034 19182 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:17.258080 19182 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:26:17.258103 19182 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:17.264492 19182 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.130:34483
I20260620 09:26:17.264530 19310 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.130:34483 every 8 connection(s)
I20260620 09:26:17.264885 19182 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781947562949448-16558-0/minicluster-data/ts-1/data/info.pb
Jun 20 09:26:17 dist-test-slave-706q krb5kdc[18941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947577, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.130@KRBTEST.COM for kudu/127.16.43.190@KRBTEST.COM
I20260620 09:26:17.272022 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 19182
I20260620 09:26:17.272128 16558 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781947562949448-16558-0/minicluster-data/ts-1/wal/instance
I20260620 09:26:17.272421 16558 external_mini_cluster.cc:1428] Setting key 1661ff7230ec6064deeb637418cd66e4
I20260620 09:26:17.274542 19029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:17.265647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.130:54883 (local address 127.16.43.190:33283)
0620 09:26:17.265689 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:17.265691 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:17.265949 (+ 258us) server_negotiation.cc:407] Connection header received
0620 09:26:17.266498 (+ 549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:17.266499 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:17.266513 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:17.266576 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:17.267553 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:17.268163 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:17.269221 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:17.269421 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:17.269496 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:17.271517 (+ 2021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:17.271519 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:17.271519 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:26:17.271559 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:17.273075 (+ 1516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:17.273560 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:17.273561 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:17.273562 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:17.273621 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:17.273852 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:17.273853 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:17.273853 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:17.274001 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:17.274069 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:17.274305 (+ 236us) server_negotiation.cc:299] Negotiation successful
0620 09:26:17.274398 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 09:26:17.274627 19313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:17.265628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:33283 (local address 127.16.43.130:54883)
0620 09:26:17.265826 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:17.265862 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:17.266381 (+ 519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:17.266661 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:17.266669 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:17.267228 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:17.267424 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:17.267428 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:17.268259 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:17.268261 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:17.269120 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:17.269122 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:17.269514 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:17.269515 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:17.269637 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:17.270103 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:26:17.270112 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:26:17.271424 (+ 1312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:26:17.273177 (+ 1753us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:17.273185 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:17.273187 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:17.273483 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:17.273691 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:17.273692 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:17.273692 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:17.273789 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:17.274061 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:26:17.274065 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:26:17.274180 (+ 115us) client_negotiation.cc:769] Sending connection context
0620 09:26:17.274262 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 09:26:17.274425 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":84,"threads_started":1}
I20260620 09:26:17.275419 19311 heartbeater.cc:347] Connected to a master server at 127.16.43.190:33283
I20260620 09:26:17.275518 19311 heartbeater.cc:464] Registering TS with master...
I20260620 09:26:17.275744 19311 heartbeater.cc:511] Master 127.16.43.190:33283 requested a full tablet report, sending...
I20260620 09:26:17.276185 18986 ts_manager.cc:194] Registered new tserver with Master: 0a7a0176b3ab461fa2cc166e2d7cb6ee (127.16.43.130:34483)
I20260620 09:26:17.276574 18986 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.43.130@KRBTEST.COM'} at 127.16.43.130:54883
WARNING: no policy specified for kudu/127.16.43.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.43.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.43.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.131.keytab.
Entry for principal kudu/127.16.43.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.131.keytab.
I20260620 09:26:17.299057 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781947562949448-16558-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781947562949448-16558-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781947562949448-16558-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781947562949448-16558-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.43.131:0
--local_ip_for_outbound_sockets=127.16.43.131
--webserver_interface=127.16.43.131
--webserver_port=0
--tserver_master_addrs=127.16.43.190:33283
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.131.keytab
--principal=kudu/127.16.43.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.43.148:41215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781947562949448-16558-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781947562949448-16558-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781947562949448-16558-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:26:17.382380 19316 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:26:17.382592 19316 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:26:17.382614 19316 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:26:17.384047 19316 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:26:17.384089 19316 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:26:17.384117 19316 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.43.131
I20260620 09:26:17.385596 19316 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:41215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781947562949448-16558-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781947562949448-16558-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.131.keytab
--principal=kudu/127.16.43.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.43.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781947562949448-16558-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.43.131
--webserver_port=0
--tserver_master_addrs=127.16.43.190:33283
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19316
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.43.131
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781947562949448-16558-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:26:17.385850 19316 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:26:17.386101 19316 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:26:17.391438 19321 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:26:17.391606 19316 server_base.cc:1034] running on GCE node
W20260620 09:26:17.391448 19322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:26:17.391470 19324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:26:17.392014 19316 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:26:17.392272 19316 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:26:17.393442 19316 hybrid_clock.cc:648] HybridClock initialized: now 1781947577393423 us; error 25 us; skew 500 ppm
Jun 20 09:26:17 dist-test-slave-706q krb5kdc[18941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947577, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:26:17.396169 19316 init.cc:377] Logged in from keytab as kudu/127.16.43.131@KRBTEST.COM (short username kudu)
I20260620 09:26:17.396713 19316 webserver.cc:460] Webserver started at http://127.16.43.131:35631/ using document root <none> and password file <none>
I20260620 09:26:17.396976 19316 fs_manager.cc:359] Metadata directory not provided
I20260620 09:26:17.397028 19316 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:26:17.397194 19316 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:26:17.398326 19316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781947562949448-16558-0/minicluster-data/ts-2/data/instance:
uuid: "d6026e4ba68a484082d41dce156a8b17"
format_stamp: "Formatted at 2026-06-20 09:26:17 on dist-test-slave-706q"
server_key: "94ce691c1e715fb1b50ec00f5adf1f8c"
server_key_iv: "2454a613d34c4f91195766eaf67a599e"
server_key_version: "encryptionkey@0"
I20260620 09:26:17.398725 19316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781947562949448-16558-0/minicluster-data/ts-2/wal/instance:
uuid: "d6026e4ba68a484082d41dce156a8b17"
format_stamp: "Formatted at 2026-06-20 09:26:17 on dist-test-slave-706q"
server_key: "94ce691c1e715fb1b50ec00f5adf1f8c"
server_key_iv: "2454a613d34c4f91195766eaf67a599e"
server_key_version: "encryptionkey@0"
I20260620 09:26:17.400281 19316 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 09:26:17.401134 19331 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:17.401348 19316 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:26:17.401432 19316 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781947562949448-16558-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781947562949448-16558-0/minicluster-data/ts-2/wal
uuid: "d6026e4ba68a484082d41dce156a8b17"
format_stamp: "Formatted at 2026-06-20 09:26:17 on dist-test-slave-706q"
server_key: "94ce691c1e715fb1b50ec00f5adf1f8c"
server_key_iv: "2454a613d34c4f91195766eaf67a599e"
server_key_version: "encryptionkey@0"
I20260620 09:26:17.401533 19316 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781947562949448-16558-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781947562949448-16558-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781947562949448-16558-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:26:17.422870 19316 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:26:17.425357 19316 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:26:17.425534 19316 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:26:17.425808 19316 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:26:17.426131 19316 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:26:17.426194 19316 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:17.426237 19316 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:26:17.426265 19316 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:17.433166 19316 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.131:34349
I20260620 09:26:17.433195 19444 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.131:34349 every 8 connection(s)
I20260620 09:26:17.433552 19316 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781947562949448-16558-0/minicluster-data/ts-2/data/info.pb
I20260620 09:26:17.434993 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 19316
I20260620 09:26:17.435134 16558 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781947562949448-16558-0/minicluster-data/ts-2/wal/instance
I20260620 09:26:17.435356 16558 external_mini_cluster.cc:1428] Setting key bee44336345b759b9f24ea2570f535a6
Jun 20 09:26:17 dist-test-slave-706q krb5kdc[18941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947577, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.131@KRBTEST.COM for kudu/127.16.43.190@KRBTEST.COM
I20260620 09:26:17.443799 19029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:17.434316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.131:37573 (local address 127.16.43.190:33283)
0620 09:26:17.434376 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:17.434379 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:17.434566 (+ 187us) server_negotiation.cc:407] Connection header received
0620 09:26:17.435099 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:17.435101 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:17.435117 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:17.435202 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:17.436484 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:17.437063 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:17.438174 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:17.438410 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:17.438475 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:17.440847 (+ 2372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:17.440848 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:17.440848 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:26:17.440855 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:17.442167 (+ 1312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:17.442596 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:17.442597 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:17.442597 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:17.442654 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:17.442947 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:17.442948 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:17.442948 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:17.443072 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:17.443181 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:17.443630 (+ 449us) server_negotiation.cc:299] Negotiation successful
0620 09:26:17.443711 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 09:26:17.443890 19447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:17.434310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:33283 (local address 127.16.43.131:37573)
0620 09:26:17.434462 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:17.434497 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:17.434973 (+ 476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:17.435300 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:17.435304 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:17.436136 (+ 832us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:17.436402 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:17.436408 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:17.437253 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:17.437254 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:17.438084 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:17.438086 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:17.438493 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:17.438494 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:17.438595 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:17.439104 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:26:17.439112 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:26:17.440738 (+ 1626us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:26:17.442246 (+ 1508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:17.442252 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:17.442254 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:17.442506 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:17.442735 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:17.442735 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:17.442735 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:17.442851 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:17.443256 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:26:17.443261 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:26:17.443385 (+ 124us) client_negotiation.cc:769] Sending connection context
0620 09:26:17.443480 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 09:26:17.443690 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":77,"threads_started":1}
I20260620 09:26:17.444443 19445 heartbeater.cc:347] Connected to a master server at 127.16.43.190:33283
I20260620 09:26:17.444563 19445 heartbeater.cc:464] Registering TS with master...
I20260620 09:26:17.444726 19445 heartbeater.cc:511] Master 127.16.43.190:33283 requested a full tablet report, sending...
I20260620 09:26:17.445127 18986 ts_manager.cc:194] Registered new tserver with Master: d6026e4ba68a484082d41dce156a8b17 (127.16.43.131:34349)
I20260620 09:26:17.445562 18986 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.43.131@KRBTEST.COM'} at 127.16.43.131:37573
I20260620 09:26:17.447257 16558 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 09:26:17.458896 19029 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:26:17.457862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41586 (local address 127.16.43.190:33283)
0620 09:26:17.457941 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:17.457944 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:17.458044 (+ 100us) server_negotiation.cc:407] Connection header received
0620 09:26:17.458132 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:17.458133 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:17.458149 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:17.458214 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:17.458801 (+ 587us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41586: BlockingRecv error: recv got EOF from 127.0.0.1:41586 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 09:26:17.459884 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 19048
I20260620 09:26:17.464768 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 19182
I20260620 09:26:17.469851 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 19316
I20260620 09:26:17.474527 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 18954
2026-06-20T09:26:17Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1872 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781947562949448-16558-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:26:17 dist-test-slave-706q krb5kdc[19457](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:26:17 dist-test-slave-706q krb5kdc[19457](info): set up 2 sockets
Jun 20 09:26:17 dist-test-slave-706q krb5kdc[19457](info): commencing operation
krb5kdc: starting...
W20260620 09:26:18.568423 16558 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781947562949448-16558-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781947562949448-16558-0/krb5kdc/test-user.keytab.
Jun 20 09:26:18 dist-test-slave-706q krb5kdc[19457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947578, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:26:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:26:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.43.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.43.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.43.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.190.keytab.
Entry for principal kudu/127.16.43.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.190.keytab.
I20260620 09:26:18.691042 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781947562949448-16558-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781947562949448-16558-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781947562949448-16558-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.43.190:42065
--webserver_interface=127.16.43.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.190.keytab
--principal=kudu/127.16.43.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.43.148:38603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.43.190:42065
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781947562949448-16558-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781947562949448-16558-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781947562949448-16558-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:26:18.772198 19470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:26:18.772480 19470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:26:18.772517 19470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:26:18.773681 19470 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:26:18.773761 19470 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:26:18.773787 19470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:26:18.773808 19470 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:26:18.773836 19470 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:26:18.775379 19470 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:38603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781947562949448-16558-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781947562949448-16558-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.43.190:42065
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.190.keytab
--principal=kudu/127.16.43.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.43.190:42065
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.43.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19470
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781947562949448-16558-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:26:18.775683 19470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:26:18.775918 19470 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:26:18.780815 19478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:26:18.780856 19476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:26:18.780833 19475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:26:18.781080 19470 server_base.cc:1034] running on GCE node
I20260620 09:26:18.781394 19470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:26:18.781713 19470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:26:18.782895 19470 hybrid_clock.cc:648] HybridClock initialized: now 1781947578782889 us; error 53 us; skew 500 ppm
Jun 20 09:26:18 dist-test-slave-706q krb5kdc[19457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947578, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:26:18.785372 19470 init.cc:377] Logged in from keytab as kudu/127.16.43.190@KRBTEST.COM (short username kudu)
I20260620 09:26:18.785872 19470 webserver.cc:460] Webserver started at http://127.16.43.190:44931/ using document root <none> and password file <none>
I20260620 09:26:18.786098 19470 fs_manager.cc:359] Metadata directory not provided
I20260620 09:26:18.786166 19470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:26:18.786301 19470 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:26:18.787393 19470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781947562949448-16558-0/minicluster-data/master-0/data/instance:
uuid: "313b0e25b2bc4ddfa399e7392f591675"
format_stamp: "Formatted at 2026-06-20 09:26:18 on dist-test-slave-706q"
server_key: "45a5a3d0ac4f5bb1d3e9d6eebd40702c"
server_key_iv: "47c8e9ebf5b1c6fe328e6c4e9002cf66"
server_key_version: "encryptionkey@0"
I20260620 09:26:18.787779 19470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781947562949448-16558-0/minicluster-data/master-0/wal/instance:
uuid: "313b0e25b2bc4ddfa399e7392f591675"
format_stamp: "Formatted at 2026-06-20 09:26:18 on dist-test-slave-706q"
server_key: "45a5a3d0ac4f5bb1d3e9d6eebd40702c"
server_key_iv: "47c8e9ebf5b1c6fe328e6c4e9002cf66"
server_key_version: "encryptionkey@0"
I20260620 09:26:18.789037 19470 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:26:18.789863 19485 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:18.790094 19470 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:26:18.790199 19470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781947562949448-16558-0/minicluster-data/master-0/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781947562949448-16558-0/minicluster-data/master-0/wal
uuid: "313b0e25b2bc4ddfa399e7392f591675"
format_stamp: "Formatted at 2026-06-20 09:26:18 on dist-test-slave-706q"
server_key: "45a5a3d0ac4f5bb1d3e9d6eebd40702c"
server_key_iv: "47c8e9ebf5b1c6fe328e6c4e9002cf66"
server_key_version: "encryptionkey@0"
I20260620 09:26:18.790298 19470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781947562949448-16558-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781947562949448-16558-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781947562949448-16558-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:26:18.806703 19470 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:26:18.808810 19470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:26:18.808987 19470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:26:18.812839 19470 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.190:42065
I20260620 09:26:18.812876 19537 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.190:42065 every 8 connection(s)
I20260620 09:26:18.813390 19470 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
I20260620 09:26:18.814150 19538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:26:18.816211 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 19470
I20260620 09:26:18.816336 16558 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781947562949448-16558-0/minicluster-data/master-0/wal/instance
I20260620 09:26:18.816527 16558 external_mini_cluster.cc:1428] Setting key 6f8f89fa8665719bf9c3fcc4976a5a06
I20260620 09:26:18.816880 19538 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 313b0e25b2bc4ddfa399e7392f591675: Bootstrap starting.
I20260620 09:26:18.817792 19538 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 313b0e25b2bc4ddfa399e7392f591675: Neither blocks nor log segments found. Creating new log.
I20260620 09:26:18.818164 19538 log.cc:824] T 00000000000000000000000000000000 P 313b0e25b2bc4ddfa399e7392f591675: Log is configured to *not* fsync() on all Append() calls
I20260620 09:26:18.819031 19538 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 313b0e25b2bc4ddfa399e7392f591675: No bootstrap required, opened a new log
I20260620 09:26:18.821432 19538 raft_consensus.cc:348] T 00000000000000000000000000000000 P 313b0e25b2bc4ddfa399e7392f591675 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "313b0e25b2bc4ddfa399e7392f591675" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 42065 } }
I20260620 09:26:18.821584 19538 raft_consensus.cc:374] T 00000000000000000000000000000000 P 313b0e25b2bc4ddfa399e7392f591675 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:26:18.821611 19538 raft_consensus.cc:729] T 00000000000000000000000000000000 P 313b0e25b2bc4ddfa399e7392f591675 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 313b0e25b2bc4ddfa399e7392f591675, State: Initialized, Role: FOLLOWER
I20260620 09:26:18.821779 19538 consensus_queue.cc:260] T 00000000000000000000000000000000 P 313b0e25b2bc4ddfa399e7392f591675 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "313b0e25b2bc4ddfa399e7392f591675" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 42065 } }
I20260620 09:26:18.821863 19538 raft_consensus.cc:388] T 00000000000000000000000000000000 P 313b0e25b2bc4ddfa399e7392f591675 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:26:18.821888 19538 raft_consensus.cc:482] T 00000000000000000000000000000000 P 313b0e25b2bc4ddfa399e7392f591675 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:26:18.821962 19538 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 313b0e25b2bc4ddfa399e7392f591675 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:26:18.822590 19538 raft_consensus.cc:504] T 00000000000000000000000000000000 P 313b0e25b2bc4ddfa399e7392f591675 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "313b0e25b2bc4ddfa399e7392f591675" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 42065 } }
I20260620 09:26:18.822727 19538 leader_election.cc:302] T 00000000000000000000000000000000 P 313b0e25b2bc4ddfa399e7392f591675 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 313b0e25b2bc4ddfa399e7392f591675; no voters:
I20260620 09:26:18.822952 19538 leader_election.cc:288] T 00000000000000000000000000000000 P 313b0e25b2bc4ddfa399e7392f591675 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:26:18.822983 19543 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 313b0e25b2bc4ddfa399e7392f591675 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:26:18.823105 19543 raft_consensus.cc:686] T 00000000000000000000000000000000 P 313b0e25b2bc4ddfa399e7392f591675 [term 1 LEADER]: Becoming Leader. State: Replica: 313b0e25b2bc4ddfa399e7392f591675, State: Running, Role: LEADER
I20260620 09:26:18.823232 19538 sys_catalog.cc:564] T 00000000000000000000000000000000 P 313b0e25b2bc4ddfa399e7392f591675 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:26:18.823215 19543 consensus_queue.cc:237] T 00000000000000000000000000000000 P 313b0e25b2bc4ddfa399e7392f591675 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "313b0e25b2bc4ddfa399e7392f591675" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 42065 } }
I20260620 09:26:18.823762 19544 sys_catalog.cc:455] T 00000000000000000000000000000000 P 313b0e25b2bc4ddfa399e7392f591675 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 313b0e25b2bc4ddfa399e7392f591675. Latest consensus state: current_term: 1 leader_uuid: "313b0e25b2bc4ddfa399e7392f591675" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "313b0e25b2bc4ddfa399e7392f591675" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 42065 } } }
I20260620 09:26:18.824004 19544 sys_catalog.cc:458] T 00000000000000000000000000000000 P 313b0e25b2bc4ddfa399e7392f591675 [sys.catalog]: This master's current role is: LEADER
I20260620 09:26:18.823918 19545 sys_catalog.cc:455] T 00000000000000000000000000000000 P 313b0e25b2bc4ddfa399e7392f591675 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "313b0e25b2bc4ddfa399e7392f591675" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "313b0e25b2bc4ddfa399e7392f591675" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 42065 } } }
I20260620 09:26:18.824203 19545 sys_catalog.cc:458] T 00000000000000000000000000000000 P 313b0e25b2bc4ddfa399e7392f591675 [sys.catalog]: This master's current role is: LEADER
I20260620 09:26:18.824411 19552 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Jun 20 09:26:18 dist-test-slave-706q krb5kdc[19457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947578, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.43.190@KRBTEST.COM
I20260620 09:26:18.824976 19552 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:26:18.826923 19552 catalog_manager.cc:1347] Generated new cluster ID: 76ff6746b70540b38f4309766f686911
I20260620 09:26:18.826989 19552 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:26:18.829543 19541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:18.817129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41010 (local address 127.16.43.190:42065)
0620 09:26:18.817373 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:18.817382 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:18.817410 (+ 28us) server_negotiation.cc:407] Connection header received
0620 09:26:18.817957 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:18.817965 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:18.818033 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:18.818163 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:18.819025 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:18.820111 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:18.821309 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:18.821870 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:18.821982 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:18.825423 (+ 3441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:18.825427 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:18.825428 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:26:18.825440 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:18.827783 (+ 2343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:18.828251 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:18.828261 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:18.828261 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:18.828368 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:18.828587 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:18.828588 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:18.828589 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:18.828904 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:18.829052 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:18.829178 (+ 126us) server_negotiation.cc:299] Negotiation successful
0620 09:26:18.829328 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":83,"threads_started":1}
I20260620 09:26:18.830665 16558 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 09:26:18.846864 19552 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:26:18.847487 19552 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:26:18.849941 19541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:18.845586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41014 (local address 127.16.43.190:42065)
0620 09:26:18.845634 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:18.845636 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:18.845649 (+ 13us) server_negotiation.cc:407] Connection header received
0620 09:26:18.845753 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:18.845754 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:18.845766 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:18.845833 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:18.846294 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:18.847056 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:18.848176 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:18.848519 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:18.848595 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:18.849401 (+ 806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:18.849403 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:18.849403 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:26:18.849638 (+ 235us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:18.849657 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:26:18.849664 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:18.849731 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:18.849809 (+ 78us) server_negotiation.cc:299] Negotiation successful
0620 09:26:18.849855 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 09:26:18.855135 19552 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 313b0e25b2bc4ddfa399e7392f591675: Generated new TSK 0
I20260620 09:26:18.855376 19552 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 09:26:18.858875 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 19470
2026-06-20T09:26:18Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1382 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781947562949448-16558-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:26:18 dist-test-slave-706q krb5kdc[19571](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:26:18 dist-test-slave-706q krb5kdc[19571](info): set up 2 sockets
Jun 20 09:26:18 dist-test-slave-706q krb5kdc[19571](info): commencing operation
krb5kdc: starting...
W20260620 09:26:19.916715 16558 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781947562949448-16558-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781947562949448-16558-0/krb5kdc/test-user.keytab.
Jun 20 09:26:19 dist-test-slave-706q krb5kdc[19571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947579, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:26:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:26:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.43.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.43.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.43.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.190.keytab.
Entry for principal kudu/127.16.43.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.190.keytab.
I20260620 09:26:20.047704 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781947562949448-16558-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781947562949448-16558-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781947562949448-16558-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.43.190:36637
--webserver_interface=127.16.43.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.190.keytab
--principal=kudu/127.16.43.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.43.148:38607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.43.190:36637
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781947562949448-16558-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781947562949448-16558-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781947562949448-16558-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:26:20.131193 19584 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:26:20.131430 19584 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:26:20.131453 19584 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:26:20.132786 19584 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:26:20.132833 19584 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:26:20.132848 19584 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:26:20.132860 19584 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:26:20.132872 19584 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:26:20.134435 19584 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:38607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781947562949448-16558-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781947562949448-16558-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.43.190:36637
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.190.keytab
--principal=kudu/127.16.43.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.43.190:36637
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.43.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19584
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781947562949448-16558-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:26:20.134713 19584 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:26:20.134958 19584 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:26:20.140010 19592 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:26:20.139984 19590 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:26:20.140177 19584 server_base.cc:1034] running on GCE node
W20260620 09:26:20.140065 19589 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:26:20.140477 19584 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:26:20.140769 19584 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:26:20.141939 19584 hybrid_clock.cc:648] HybridClock initialized: now 1781947580141915 us; error 50 us; skew 500 ppm
Jun 20 09:26:20 dist-test-slave-706q krb5kdc[19571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947580, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:26:20.145083 19584 init.cc:377] Logged in from keytab as kudu/127.16.43.190@KRBTEST.COM (short username kudu)
I20260620 09:26:20.145591 19584 webserver.cc:460] Webserver started at http://127.16.43.190:46625/ using document root <none> and password file <none>
I20260620 09:26:20.145843 19584 fs_manager.cc:359] Metadata directory not provided
I20260620 09:26:20.145891 19584 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:26:20.146026 19584 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:26:20.147094 19584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781947562949448-16558-0/minicluster-data/master-0/data/instance:
uuid: "fe707619d3f34f6ba127f135fc95114c"
format_stamp: "Formatted at 2026-06-20 09:26:20 on dist-test-slave-706q"
server_key: "5bd8ec764196ddaef178c7ecad8f64a6"
server_key_iv: "c4ca95c2427eee2252d7a70368d80293"
server_key_version: "encryptionkey@0"
I20260620 09:26:20.147454 19584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781947562949448-16558-0/minicluster-data/master-0/wal/instance:
uuid: "fe707619d3f34f6ba127f135fc95114c"
format_stamp: "Formatted at 2026-06-20 09:26:20 on dist-test-slave-706q"
server_key: "5bd8ec764196ddaef178c7ecad8f64a6"
server_key_iv: "c4ca95c2427eee2252d7a70368d80293"
server_key_version: "encryptionkey@0"
I20260620 09:26:20.148852 19584 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 09:26:20.149701 19599 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:20.149940 19584 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 09:26:20.150018 19584 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781947562949448-16558-0/minicluster-data/master-0/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781947562949448-16558-0/minicluster-data/master-0/wal
uuid: "fe707619d3f34f6ba127f135fc95114c"
format_stamp: "Formatted at 2026-06-20 09:26:20 on dist-test-slave-706q"
server_key: "5bd8ec764196ddaef178c7ecad8f64a6"
server_key_iv: "c4ca95c2427eee2252d7a70368d80293"
server_key_version: "encryptionkey@0"
I20260620 09:26:20.150112 19584 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781947562949448-16558-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781947562949448-16558-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781947562949448-16558-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:26:20.173171 19584 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:26:20.175391 19584 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:26:20.175513 19584 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:26:20.179678 19584 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.190:36637
I20260620 09:26:20.179715 19651 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.190:36637 every 8 connection(s)
I20260620 09:26:20.180117 19584 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
I20260620 09:26:20.181197 19652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:26:20.182977 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 19584
I20260620 09:26:20.183116 16558 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781947562949448-16558-0/minicluster-data/master-0/wal/instance
I20260620 09:26:20.183284 16558 external_mini_cluster.cc:1428] Setting key 71f2c65c6bbcf784db52edc687a54e8c
I20260620 09:26:20.183897 19652 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe707619d3f34f6ba127f135fc95114c: Bootstrap starting.
I20260620 09:26:20.184532 19652 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fe707619d3f34f6ba127f135fc95114c: Neither blocks nor log segments found. Creating new log.
I20260620 09:26:20.184881 19652 log.cc:824] T 00000000000000000000000000000000 P fe707619d3f34f6ba127f135fc95114c: Log is configured to *not* fsync() on all Append() calls
I20260620 09:26:20.185777 19652 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe707619d3f34f6ba127f135fc95114c: No bootstrap required, opened a new log
I20260620 09:26:20.187986 19652 raft_consensus.cc:348] T 00000000000000000000000000000000 P fe707619d3f34f6ba127f135fc95114c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe707619d3f34f6ba127f135fc95114c" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 36637 } }
I20260620 09:26:20.188144 19652 raft_consensus.cc:374] T 00000000000000000000000000000000 P fe707619d3f34f6ba127f135fc95114c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:26:20.188206 19652 raft_consensus.cc:729] T 00000000000000000000000000000000 P fe707619d3f34f6ba127f135fc95114c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe707619d3f34f6ba127f135fc95114c, State: Initialized, Role: FOLLOWER
I20260620 09:26:20.188377 19652 consensus_queue.cc:260] T 00000000000000000000000000000000 P fe707619d3f34f6ba127f135fc95114c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe707619d3f34f6ba127f135fc95114c" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 36637 } }
I20260620 09:26:20.188490 19652 raft_consensus.cc:388] T 00000000000000000000000000000000 P fe707619d3f34f6ba127f135fc95114c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:26:20.188539 19652 raft_consensus.cc:482] T 00000000000000000000000000000000 P fe707619d3f34f6ba127f135fc95114c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:26:20.188895 19652 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fe707619d3f34f6ba127f135fc95114c [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:26:20.189472 19652 raft_consensus.cc:504] T 00000000000000000000000000000000 P fe707619d3f34f6ba127f135fc95114c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe707619d3f34f6ba127f135fc95114c" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 36637 } }
I20260620 09:26:20.189596 19652 leader_election.cc:302] T 00000000000000000000000000000000 P fe707619d3f34f6ba127f135fc95114c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fe707619d3f34f6ba127f135fc95114c; no voters:
I20260620 09:26:20.189800 19652 leader_election.cc:288] T 00000000000000000000000000000000 P fe707619d3f34f6ba127f135fc95114c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:26:20.189822 19657 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fe707619d3f34f6ba127f135fc95114c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:26:20.189939 19657 raft_consensus.cc:686] T 00000000000000000000000000000000 P fe707619d3f34f6ba127f135fc95114c [term 1 LEADER]: Becoming Leader. State: Replica: fe707619d3f34f6ba127f135fc95114c, State: Running, Role: LEADER
I20260620 09:26:20.190078 19657 consensus_queue.cc:237] T 00000000000000000000000000000000 P fe707619d3f34f6ba127f135fc95114c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe707619d3f34f6ba127f135fc95114c" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 36637 } }
I20260620 09:26:20.190078 19652 sys_catalog.cc:564] T 00000000000000000000000000000000 P fe707619d3f34f6ba127f135fc95114c [sys.catalog]: configured and running, proceeding with master startup.
Jun 20 09:26:20 dist-test-slave-706q krb5kdc[19571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947579, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.43.190@KRBTEST.COM
I20260620 09:26:20.190503 19659 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe707619d3f34f6ba127f135fc95114c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fe707619d3f34f6ba127f135fc95114c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe707619d3f34f6ba127f135fc95114c" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 36637 } } }
I20260620 09:26:20.190572 19659 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe707619d3f34f6ba127f135fc95114c [sys.catalog]: This master's current role is: LEADER
I20260620 09:26:20.190527 19658 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe707619d3f34f6ba127f135fc95114c [sys.catalog]: SysCatalogTable state changed. Reason: New leader fe707619d3f34f6ba127f135fc95114c. Latest consensus state: current_term: 1 leader_uuid: "fe707619d3f34f6ba127f135fc95114c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe707619d3f34f6ba127f135fc95114c" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 36637 } } }
I20260620 09:26:20.190616 19658 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe707619d3f34f6ba127f135fc95114c [sys.catalog]: This master's current role is: LEADER
I20260620 09:26:20.191375 19661 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:26:20.192065 19661 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:26:20.194041 19661 catalog_manager.cc:1347] Generated new cluster ID: 5705c18c9c2d4f258471fa8aa12cb25c
I20260620 09:26:20.194105 19661 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:26:20.195511 19655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:20.183946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41564 (local address 127.16.43.190:36637)
0620 09:26:20.184130 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:20.184138 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:20.184164 (+ 26us) server_negotiation.cc:407] Connection header received
0620 09:26:20.184588 (+ 424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:20.184595 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:20.184671 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:20.184807 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:20.185552 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:20.186525 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:20.187689 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:20.188118 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:20.188280 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:20.191227 (+ 2947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:20.191232 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:20.191232 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:26:20.191247 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:20.193402 (+ 2155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:20.193873 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:20.193876 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:20.193876 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:20.193982 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:20.194235 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:20.194236 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:20.194236 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:20.194517 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:20.194650 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:20.195166 (+ 516us) server_negotiation.cc:299] Negotiation successful
0620 09:26:20.195303 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":94,"threads_started":1}
I20260620 09:26:20.198904 19661 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:26:20.199450 19661 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:26:20.202234 19661 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fe707619d3f34f6ba127f135fc95114c: Generated new TSK 0
I20260620 09:26:20.202409 19661 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.43.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.43.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.43.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.129.keytab.
Entry for principal kudu/127.16.43.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.129.keytab.
I20260620 09:26:20.230351 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781947562949448-16558-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781947562949448-16558-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781947562949448-16558-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781947562949448-16558-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.43.129:0
--local_ip_for_outbound_sockets=127.16.43.129
--webserver_interface=127.16.43.129
--webserver_port=0
--tserver_master_addrs=127.16.43.190:36637
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.129.keytab
--principal=kudu/127.16.43.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.43.148:38607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781947562949448-16558-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781947562949448-16558-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781947562949448-16558-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:26:20.324558 19678 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:26:20.324766 19678 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:26:20.324790 19678 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:26:20.326017 19678 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:26:20.326063 19678 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:26:20.326090 19678 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.43.129
I20260620 09:26:20.327692 19678 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:38607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781947562949448-16558-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781947562949448-16558-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.129.keytab
--principal=kudu/127.16.43.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.43.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781947562949448-16558-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.43.129
--webserver_port=0
--tserver_master_addrs=127.16.43.190:36637
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19678
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.43.129
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781947562949448-16558-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:26:20.327950 19678 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:26:20.328212 19678 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:26:20.333858 19683 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:26:20.333967 19684 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:26:20.333861 19686 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:26:20.334105 19678 server_base.cc:1034] running on GCE node
I20260620 09:26:20.334384 19678 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:26:20.334712 19678 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:26:20.335882 19678 hybrid_clock.cc:648] HybridClock initialized: now 1781947580335869 us; error 33 us; skew 500 ppm
Jun 20 09:26:20 dist-test-slave-706q krb5kdc[19571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947580, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:26:20.338739 19678 init.cc:377] Logged in from keytab as kudu/127.16.43.129@KRBTEST.COM (short username kudu)
I20260620 09:26:20.339262 19678 webserver.cc:460] Webserver started at http://127.16.43.129:45581/ using document root <none> and password file <none>
I20260620 09:26:20.339576 19678 fs_manager.cc:359] Metadata directory not provided
I20260620 09:26:20.339656 19678 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:26:20.339821 19678 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:26:20.341328 19678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781947562949448-16558-0/minicluster-data/ts-0/data/instance:
uuid: "8a3b2c666f02429f869c7952458f2246"
format_stamp: "Formatted at 2026-06-20 09:26:20 on dist-test-slave-706q"
server_key: "885ef1ce82107a98517cc530c682a678"
server_key_iv: "7def5b311f52f68659978526a64753fd"
server_key_version: "encryptionkey@0"
I20260620 09:26:20.341709 19678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781947562949448-16558-0/minicluster-data/ts-0/wal/instance:
uuid: "8a3b2c666f02429f869c7952458f2246"
format_stamp: "Formatted at 2026-06-20 09:26:20 on dist-test-slave-706q"
server_key: "885ef1ce82107a98517cc530c682a678"
server_key_iv: "7def5b311f52f68659978526a64753fd"
server_key_version: "encryptionkey@0"
I20260620 09:26:20.343129 19678 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 09:26:20.343971 19693 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:20.344151 19678 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:20.344256 19678 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781947562949448-16558-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781947562949448-16558-0/minicluster-data/ts-0/wal
uuid: "8a3b2c666f02429f869c7952458f2246"
format_stamp: "Formatted at 2026-06-20 09:26:20 on dist-test-slave-706q"
server_key: "885ef1ce82107a98517cc530c682a678"
server_key_iv: "7def5b311f52f68659978526a64753fd"
server_key_version: "encryptionkey@0"
I20260620 09:26:20.344331 19678 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781947562949448-16558-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781947562949448-16558-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781947562949448-16558-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:26:20.354179 19678 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:26:20.356623 19678 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:26:20.356798 19678 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:26:20.357052 19678 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:26:20.357378 19678 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:26:20.357434 19678 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:20.357477 19678 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:26:20.357517 19678 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:20.363854 19678 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.129:33795
I20260620 09:26:20.363904 19806 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.129:33795 every 8 connection(s)
I20260620 09:26:20.364243 19678 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781947562949448-16558-0/minicluster-data/ts-0/data/info.pb
Jun 20 09:26:20 dist-test-slave-706q krb5kdc[19571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947580, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.129@KRBTEST.COM for kudu/127.16.43.190@KRBTEST.COM
I20260620 09:26:20.370872 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 19678
I20260620 09:26:20.370986 16558 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781947562949448-16558-0/minicluster-data/ts-0/wal/instance
I20260620 09:26:20.371238 16558 external_mini_cluster.cc:1428] Setting key a274dbe4a83a50b27b56ef1aeca88c52
I20260620 09:26:20.374619 19655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:20.365044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.129:52857 (local address 127.16.43.190:36637)
0620 09:26:20.365103 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:20.365105 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:20.365296 (+ 191us) server_negotiation.cc:407] Connection header received
0620 09:26:20.365865 (+ 569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:20.365867 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:20.365880 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:20.365934 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:20.366972 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:20.367599 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:20.368677 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:20.368890 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:20.368998 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:20.371550 (+ 2552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:20.371552 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:20.371553 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:26:20.371562 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:20.373044 (+ 1482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:20.373590 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:20.373591 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:20.373592 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:20.373663 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:20.373910 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:20.373911 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:20.373912 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:20.374077 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:20.374147 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:20.374408 (+ 261us) server_negotiation.cc:299] Negotiation successful
0620 09:26:20.374509 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 09:26:20.374919 19809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:20.364985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:36637 (local address 127.16.43.129:52857)
0620 09:26:20.365148 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:20.365202 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:20.365746 (+ 544us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:20.366029 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:20.366039 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:20.366636 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:20.366851 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:20.366856 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:20.367702 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:20.367703 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:20.368559 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:20.368561 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:20.369046 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:20.369047 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:20.369187 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:20.369746 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:26:20.369755 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:26:20.371419 (+ 1664us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:26:20.373131 (+ 1712us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:20.373138 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:20.373142 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:20.373499 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:20.373737 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:20.373738 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:20.373738 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:20.373850 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:20.374142 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:26:20.374146 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:26:20.374277 (+ 131us) client_negotiation.cc:769] Sending connection context
0620 09:26:20.374363 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 09:26:20.374620 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":82,"threads_started":1}
I20260620 09:26:20.375563 19807 heartbeater.cc:347] Connected to a master server at 127.16.43.190:36637
I20260620 09:26:20.375680 19807 heartbeater.cc:464] Registering TS with master...
I20260620 09:26:20.375916 19807 heartbeater.cc:511] Master 127.16.43.190:36637 requested a full tablet report, sending...
I20260620 09:26:20.376535 19616 ts_manager.cc:194] Registered new tserver with Master: 8a3b2c666f02429f869c7952458f2246 (127.16.43.129:33795)
I20260620 09:26:20.377205 19616 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.43.129@KRBTEST.COM'} at 127.16.43.129:52857
WARNING: no policy specified for kudu/127.16.43.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.43.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.43.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.130.keytab.
Entry for principal kudu/127.16.43.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.130.keytab.
I20260620 09:26:20.398613 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781947562949448-16558-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781947562949448-16558-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781947562949448-16558-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781947562949448-16558-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.43.130:0
--local_ip_for_outbound_sockets=127.16.43.130
--webserver_interface=127.16.43.130
--webserver_port=0
--tserver_master_addrs=127.16.43.190:36637
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.130.keytab
--principal=kudu/127.16.43.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.43.148:38607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781947562949448-16558-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781947562949448-16558-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781947562949448-16558-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:26:20.484143 19812 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:26:20.484366 19812 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:26:20.484393 19812 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:26:20.485661 19812 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:26:20.485711 19812 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:26:20.485740 19812 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.43.130
I20260620 09:26:20.487258 19812 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:38607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781947562949448-16558-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781947562949448-16558-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.130.keytab
--principal=kudu/127.16.43.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.43.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781947562949448-16558-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.43.130
--webserver_port=0
--tserver_master_addrs=127.16.43.190:36637
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19812
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.43.130
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781947562949448-16558-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:26:20.487517 19812 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:26:20.487808 19812 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:26:20.493297 19818 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:26:20.493348 19812 server_base.cc:1034] running on GCE node
W20260620 09:26:20.493319 19817 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:26:20.493319 19820 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:26:20.493868 19812 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:26:20.494143 19812 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:26:20.495296 19812 hybrid_clock.cc:648] HybridClock initialized: now 1781947580495283 us; error 29 us; skew 500 ppm
Jun 20 09:26:20 dist-test-slave-706q krb5kdc[19571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947580, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:26:20.497773 19812 init.cc:377] Logged in from keytab as kudu/127.16.43.130@KRBTEST.COM (short username kudu)
I20260620 09:26:20.498242 19812 webserver.cc:460] Webserver started at http://127.16.43.130:37331/ using document root <none> and password file <none>
I20260620 09:26:20.498474 19812 fs_manager.cc:359] Metadata directory not provided
I20260620 09:26:20.498541 19812 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:26:20.498694 19812 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:26:20.499866 19812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781947562949448-16558-0/minicluster-data/ts-1/data/instance:
uuid: "3e23fe2c321d44be8dd1227f3d6cc1ab"
format_stamp: "Formatted at 2026-06-20 09:26:20 on dist-test-slave-706q"
server_key: "456abf5194761080742bac79f1b8b44e"
server_key_iv: "a2525d23b1e8f0f137033f68b74160ce"
server_key_version: "encryptionkey@0"
I20260620 09:26:20.500241 19812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781947562949448-16558-0/minicluster-data/ts-1/wal/instance:
uuid: "3e23fe2c321d44be8dd1227f3d6cc1ab"
format_stamp: "Formatted at 2026-06-20 09:26:20 on dist-test-slave-706q"
server_key: "456abf5194761080742bac79f1b8b44e"
server_key_iv: "a2525d23b1e8f0f137033f68b74160ce"
server_key_version: "encryptionkey@0"
I20260620 09:26:20.501643 19812 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 09:26:20.502478 19827 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:20.502775 19812 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:26:20.502882 19812 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781947562949448-16558-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781947562949448-16558-0/minicluster-data/ts-1/wal
uuid: "3e23fe2c321d44be8dd1227f3d6cc1ab"
format_stamp: "Formatted at 2026-06-20 09:26:20 on dist-test-slave-706q"
server_key: "456abf5194761080742bac79f1b8b44e"
server_key_iv: "a2525d23b1e8f0f137033f68b74160ce"
server_key_version: "encryptionkey@0"
I20260620 09:26:20.502970 19812 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781947562949448-16558-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781947562949448-16558-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781947562949448-16558-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:26:20.514081 19812 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:26:20.516470 19812 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:26:20.516646 19812 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:26:20.516901 19812 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:26:20.517233 19812 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:26:20.517290 19812 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:20.517328 19812 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:26:20.517376 19812 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:20.523322 19812 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.130:45717
I20260620 09:26:20.523406 19940 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.130:45717 every 8 connection(s)
I20260620 09:26:20.523785 19812 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781947562949448-16558-0/minicluster-data/ts-1/data/info.pb
I20260620 09:26:20.524345 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 19812
I20260620 09:26:20.524466 16558 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781947562949448-16558-0/minicluster-data/ts-1/wal/instance
I20260620 09:26:20.524674 16558 external_mini_cluster.cc:1428] Setting key 6f40957bbe5c3aaa5e018653db929e64
Jun 20 09:26:20 dist-test-slave-706q krb5kdc[19571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947580, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.130@KRBTEST.COM for kudu/127.16.43.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.43.131@KRBTEST.COM; defaulting to no policy
I20260620 09:26:20.534988 19655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:20.524562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.130:51883 (local address 127.16.43.190:36637)
0620 09:26:20.524615 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:20.524617 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:20.524901 (+ 284us) server_negotiation.cc:407] Connection header received
0620 09:26:20.525504 (+ 603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:20.525509 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:20.525521 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:20.525561 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:20.526988 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:20.527866 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:20.528962 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:20.529320 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:20.529398 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:20.531906 (+ 2508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:20.531908 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:20.531908 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:26:20.531916 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:20.533569 (+ 1653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:20.533948 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:20.533949 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:20.533949 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:20.533998 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:20.534216 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:20.534217 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:20.534217 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:20.534316 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:20.534434 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:20.534786 (+ 352us) server_negotiation.cc:299] Negotiation successful
0620 09:26:20.534863 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 09:26:20.535157 19943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:20.524564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:36637 (local address 127.16.43.130:51883)
0620 09:26:20.524750 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:20.524804 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:20.525386 (+ 582us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:20.525645 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:20.525650 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:20.526542 (+ 892us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:20.526854 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:20.526862 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:20.528004 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:20.528005 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:20.528858 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:20.528860 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:20.529415 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:20.529416 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:20.529523 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:20.530260 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:26:20.530270 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:26:20.531809 (+ 1539us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:26:20.533646 (+ 1837us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:20.533652 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:20.533654 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:20.533871 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:20.534060 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:20.534060 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:20.534061 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:20.534159 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:20.534434 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:26:20.534500 (+ 66us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:26:20.534633 (+ 133us) client_negotiation.cc:769] Sending connection context
0620 09:26:20.534751 (+ 118us) client_negotiation.cc:240] Negotiation successful
0620 09:26:20.534916 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":81,"threads_started":1}
I20260620 09:26:20.535776 19941 heartbeater.cc:347] Connected to a master server at 127.16.43.190:36637
I20260620 09:26:20.535907 19941 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.43.131@KRBTEST.COM" created.
I20260620 09:26:20.536141 19941 heartbeater.cc:511] Master 127.16.43.190:36637 requested a full tablet report, sending...
I20260620 09:26:20.536640 19616 ts_manager.cc:194] Registered new tserver with Master: 3e23fe2c321d44be8dd1227f3d6cc1ab (127.16.43.130:45717)
I20260620 09:26:20.537164 19616 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.43.130@KRBTEST.COM'} at 127.16.43.130:51883
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.43.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.131.keytab.
Entry for principal kudu/127.16.43.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.131.keytab.
I20260620 09:26:20.552342 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781947562949448-16558-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781947562949448-16558-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781947562949448-16558-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781947562949448-16558-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.43.131:0
--local_ip_for_outbound_sockets=127.16.43.131
--webserver_interface=127.16.43.131
--webserver_port=0
--tserver_master_addrs=127.16.43.190:36637
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.131.keytab
--principal=kudu/127.16.43.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.43.148:38607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781947562949448-16558-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781947562949448-16558-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781947562949448-16558-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:26:20.639372 19946 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:26:20.639629 19946 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:26:20.639655 19946 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:26:20.640868 19946 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:26:20.640915 19946 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:26:20.640942 19946 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.43.131
I20260620 09:26:20.642426 19946 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:38607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781947562949448-16558-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781947562949448-16558-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.131.keytab
--principal=kudu/127.16.43.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.43.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781947562949448-16558-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.43.131
--webserver_port=0
--tserver_master_addrs=127.16.43.190:36637
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19946
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.43.131
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781947562949448-16558-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:26:20.642684 19946 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:26:20.642956 19946 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:26:20.648181 19952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:26:20.648239 19954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:26:20.648200 19951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:26:20.648425 19946 server_base.cc:1034] running on GCE node
I20260620 09:26:20.648687 19946 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:26:20.648964 19946 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:26:20.650118 19946 hybrid_clock.cc:648] HybridClock initialized: now 1781947580650108 us; error 52 us; skew 500 ppm
Jun 20 09:26:20 dist-test-slave-706q krb5kdc[19571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947580, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:26:20.652917 19946 init.cc:377] Logged in from keytab as kudu/127.16.43.131@KRBTEST.COM (short username kudu)
I20260620 09:26:20.653438 19946 webserver.cc:460] Webserver started at http://127.16.43.131:39665/ using document root <none> and password file <none>
I20260620 09:26:20.653687 19946 fs_manager.cc:359] Metadata directory not provided
I20260620 09:26:20.653753 19946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:26:20.653915 19946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:26:20.654986 19946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781947562949448-16558-0/minicluster-data/ts-2/data/instance:
uuid: "b2d81284d9214cdda7592d6e15b9297b"
format_stamp: "Formatted at 2026-06-20 09:26:20 on dist-test-slave-706q"
server_key: "f4dc0d55937e740805ead2afbeb0d482"
server_key_iv: "0b0ccb60221ad2a50784c9c81f66c609"
server_key_version: "encryptionkey@0"
I20260620 09:26:20.655357 19946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781947562949448-16558-0/minicluster-data/ts-2/wal/instance:
uuid: "b2d81284d9214cdda7592d6e15b9297b"
format_stamp: "Formatted at 2026-06-20 09:26:20 on dist-test-slave-706q"
server_key: "f4dc0d55937e740805ead2afbeb0d482"
server_key_iv: "0b0ccb60221ad2a50784c9c81f66c609"
server_key_version: "encryptionkey@0"
I20260620 09:26:20.656785 19946 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 09:26:20.657672 19961 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:20.657964 19946 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:26:20.658051 19946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781947562949448-16558-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781947562949448-16558-0/minicluster-data/ts-2/wal
uuid: "b2d81284d9214cdda7592d6e15b9297b"
format_stamp: "Formatted at 2026-06-20 09:26:20 on dist-test-slave-706q"
server_key: "f4dc0d55937e740805ead2afbeb0d482"
server_key_iv: "0b0ccb60221ad2a50784c9c81f66c609"
server_key_version: "encryptionkey@0"
I20260620 09:26:20.658146 19946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781947562949448-16558-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781947562949448-16558-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781947562949448-16558-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:26:20.672472 19946 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:26:20.674695 19946 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:26:20.674893 19946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:26:20.675156 19946 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:26:20.675508 19946 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:26:20.675635 19946 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:20.675678 19946 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:26:20.675704 19946 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:20.682075 19946 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.131:36355
I20260620 09:26:20.682119 20074 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.131:36355 every 8 connection(s)
I20260620 09:26:20.682453 19946 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781947562949448-16558-0/minicluster-data/ts-2/data/info.pb
I20260620 09:26:20.687737 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 19946
I20260620 09:26:20.687860 16558 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781947562949448-16558-0/minicluster-data/ts-2/wal/instance
I20260620 09:26:20.688055 16558 external_mini_cluster.cc:1428] Setting key def6277fb9545e222fc0f885949afea8
Jun 20 09:26:20 dist-test-slave-706q krb5kdc[19571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947580, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.131@KRBTEST.COM for kudu/127.16.43.190@KRBTEST.COM
I20260620 09:26:20.692914 19655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:20.683322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.131:36083 (local address 127.16.43.190:36637)
0620 09:26:20.683386 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:20.683388 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:20.683599 (+ 211us) server_negotiation.cc:407] Connection header received
0620 09:26:20.684128 (+ 529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:20.684129 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:20.684140 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:20.684211 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:20.685178 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:20.685729 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:20.686887 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:20.687084 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:20.687152 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:20.689794 (+ 2642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:20.689795 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:20.689796 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:26:20.689806 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:20.691317 (+ 1511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:20.691890 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:20.691892 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:20.691892 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:20.691955 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:20.692229 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:20.692230 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:20.692231 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:20.692359 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:20.692437 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:20.692727 (+ 290us) server_negotiation.cc:299] Negotiation successful
0620 09:26:20.692810 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 09:26:20.693133 20077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:20.683297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:36637 (local address 127.16.43.131:36083)
0620 09:26:20.683459 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:20.683501 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:20.684028 (+ 527us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:20.684267 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:20.684270 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:20.684841 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:20.685062 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:20.685068 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:20.685840 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:20.685841 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:20.686770 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:20.686773 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:20.687171 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:20.687172 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:20.687283 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:20.688026 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:26:20.688035 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:26:20.689697 (+ 1662us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:26:20.691467 (+ 1770us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:20.691474 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:20.691476 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:20.691793 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:20.692034 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:20.692035 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:20.692036 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:20.692156 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:20.692427 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:26:20.692434 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:26:20.692586 (+ 152us) client_negotiation.cc:769] Sending connection context
0620 09:26:20.692694 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 09:26:20.692866 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":91,"threads_started":1}
I20260620 09:26:20.693646 20075 heartbeater.cc:347] Connected to a master server at 127.16.43.190:36637
I20260620 09:26:20.693754 20075 heartbeater.cc:464] Registering TS with master...
I20260620 09:26:20.693940 20075 heartbeater.cc:511] Master 127.16.43.190:36637 requested a full tablet report, sending...
I20260620 09:26:20.694314 19616 ts_manager.cc:194] Registered new tserver with Master: b2d81284d9214cdda7592d6e15b9297b (127.16.43.131:36355)
I20260620 09:26:20.694734 19616 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.43.131@KRBTEST.COM'} at 127.16.43.131:36083
I20260620 09:26:20.700488 16558 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 09:26:20 dist-test-slave-706q krb5kdc[19571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947580, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 20 09:26:20 dist-test-slave-706q krb5kdc[19571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947580, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.16.43.190@KRBTEST.COM
I20260620 09:26:20.724886 19655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:20.717172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41572 (local address 127.16.43.190:36637)
0620 09:26:20.717262 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:20.717264 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:20.717370 (+ 106us) server_negotiation.cc:407] Connection header received
0620 09:26:20.717426 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:20.717427 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:20.717438 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:20.717484 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:20.718102 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:20.718654 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:20.719500 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:20.719771 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:20.719830 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:20.722208 (+ 2378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:20.722210 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:20.722210 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:26:20.722218 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:20.723686 (+ 1468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:20.724045 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:20.724046 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:20.724047 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:20.724101 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:20.724373 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:20.724374 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:20.724374 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:20.724529 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:20.724608 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:20.724740 (+ 132us) server_negotiation.cc:299] Negotiation successful
0620 09:26:20.724813 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260620 09:26:20.725170 19616 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:41572
I20260620 09:26:20.725628 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 19678
I20260620 09:26:20.730165 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 19812
I20260620 09:26:20.734615 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 19946
I20260620 09:26:20.739609 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 19584
2026-06-20T09:26:20Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1882 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781947562949448-16558-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:26:20 dist-test-slave-706q krb5kdc[20087](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:26:20 dist-test-slave-706q krb5kdc[20087](info): set up 2 sockets
Jun 20 09:26:20 dist-test-slave-706q krb5kdc[20087](info): commencing operation
krb5kdc: starting...
W20260620 09:26:21.809098 16558 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781947562949448-16558-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781947562949448-16558-0/krb5kdc/test-user.keytab.
Jun 20 09:26:21 dist-test-slave-706q krb5kdc[20087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947581, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:26:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:26:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.43.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.43.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.43.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.190.keytab.
Entry for principal kudu/127.16.43.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.190.keytab.
I20260620 09:26:21.934680 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781947562949448-16558-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781947562949448-16558-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781947562949448-16558-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.43.190:45569
--webserver_interface=127.16.43.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.190.keytab
--principal=kudu/127.16.43.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.43.148:35451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.43.190:45569
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781947562949448-16558-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781947562949448-16558-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781947562949448-16558-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:26:22.016386 20100 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:26:22.016654 20100 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:26:22.016680 20100 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:26:22.017910 20100 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:26:22.017956 20100 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:26:22.017969 20100 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:26:22.017984 20100 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:26:22.017997 20100 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:26:22.019567 20100 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:35451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781947562949448-16558-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781947562949448-16558-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.43.190:45569
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.190.keytab
--principal=kudu/127.16.43.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.43.190:45569
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.43.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20100
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781947562949448-16558-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:26:22.019824 20100 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:26:22.020083 20100 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:26:22.025223 20106 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:26:22.025223 20105 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:26:22.025223 20108 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:26:22.025326 20100 server_base.cc:1034] running on GCE node
I20260620 09:26:22.025712 20100 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:26:22.026026 20100 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:26:22.027185 20100 hybrid_clock.cc:648] HybridClock initialized: now 1781947582027169 us; error 31 us; skew 500 ppm
Jun 20 09:26:22 dist-test-slave-706q krb5kdc[20087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947582, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:26:22.034267 20100 init.cc:377] Logged in from keytab as kudu/127.16.43.190@KRBTEST.COM (short username kudu)
I20260620 09:26:22.034854 20100 webserver.cc:460] Webserver started at http://127.16.43.190:40039/ using document root <none> and password file <none>
I20260620 09:26:22.035142 20100 fs_manager.cc:359] Metadata directory not provided
I20260620 09:26:22.035193 20100 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:26:22.035394 20100 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:26:22.036690 20100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781947562949448-16558-0/minicluster-data/master-0/data/instance:
uuid: "c4b578f30e784ce18d310298599537de"
format_stamp: "Formatted at 2026-06-20 09:26:22 on dist-test-slave-706q"
server_key: "1a658f5dd49fd6d0ac6ea5483397ca49"
server_key_iv: "669afa800bb8b6cde416ba2c6811fe18"
server_key_version: "encryptionkey@0"
I20260620 09:26:22.037144 20100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781947562949448-16558-0/minicluster-data/master-0/wal/instance:
uuid: "c4b578f30e784ce18d310298599537de"
format_stamp: "Formatted at 2026-06-20 09:26:22 on dist-test-slave-706q"
server_key: "1a658f5dd49fd6d0ac6ea5483397ca49"
server_key_iv: "669afa800bb8b6cde416ba2c6811fe18"
server_key_version: "encryptionkey@0"
I20260620 09:26:22.039124 20100 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20260620 09:26:22.040159 20115 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:22.040324 20100 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 09:26:22.040429 20100 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781947562949448-16558-0/minicluster-data/master-0/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781947562949448-16558-0/minicluster-data/master-0/wal
uuid: "c4b578f30e784ce18d310298599537de"
format_stamp: "Formatted at 2026-06-20 09:26:22 on dist-test-slave-706q"
server_key: "1a658f5dd49fd6d0ac6ea5483397ca49"
server_key_iv: "669afa800bb8b6cde416ba2c6811fe18"
server_key_version: "encryptionkey@0"
I20260620 09:26:22.040520 20100 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781947562949448-16558-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781947562949448-16558-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781947562949448-16558-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:26:22.068194 20100 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:26:22.070531 20100 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:26:22.070734 20100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:26:22.074747 20100 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.190:45569
I20260620 09:26:22.074780 20167 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.190:45569 every 8 connection(s)
I20260620 09:26:22.075322 20100 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
I20260620 09:26:22.076056 20168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:26:22.078297 20168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4b578f30e784ce18d310298599537de: Bootstrap starting.
I20260620 09:26:22.078835 20168 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c4b578f30e784ce18d310298599537de: Neither blocks nor log segments found. Creating new log.
I20260620 09:26:22.079074 20168 log.cc:824] T 00000000000000000000000000000000 P c4b578f30e784ce18d310298599537de: Log is configured to *not* fsync() on all Append() calls
I20260620 09:26:22.079761 20168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4b578f30e784ce18d310298599537de: No bootstrap required, opened a new log
I20260620 09:26:22.080044 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 20100
I20260620 09:26:22.080145 16558 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781947562949448-16558-0/minicluster-data/master-0/wal/instance
I20260620 09:26:22.080322 16558 external_mini_cluster.cc:1428] Setting key 304fa577feb5fcfa86448f6219bde063
I20260620 09:26:22.082281 20168 raft_consensus.cc:348] T 00000000000000000000000000000000 P c4b578f30e784ce18d310298599537de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4b578f30e784ce18d310298599537de" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 45569 } }
I20260620 09:26:22.082413 20168 raft_consensus.cc:374] T 00000000000000000000000000000000 P c4b578f30e784ce18d310298599537de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:26:22.082435 20168 raft_consensus.cc:729] T 00000000000000000000000000000000 P c4b578f30e784ce18d310298599537de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4b578f30e784ce18d310298599537de, State: Initialized, Role: FOLLOWER
I20260620 09:26:22.082594 20168 consensus_queue.cc:260] T 00000000000000000000000000000000 P c4b578f30e784ce18d310298599537de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4b578f30e784ce18d310298599537de" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 45569 } }
I20260620 09:26:22.082696 20168 raft_consensus.cc:388] T 00000000000000000000000000000000 P c4b578f30e784ce18d310298599537de [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:26:22.082742 20168 raft_consensus.cc:482] T 00000000000000000000000000000000 P c4b578f30e784ce18d310298599537de [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:26:22.082790 20168 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c4b578f30e784ce18d310298599537de [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:26:22.083287 20168 raft_consensus.cc:504] T 00000000000000000000000000000000 P c4b578f30e784ce18d310298599537de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4b578f30e784ce18d310298599537de" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 45569 } }
I20260620 09:26:22.083405 20168 leader_election.cc:302] T 00000000000000000000000000000000 P c4b578f30e784ce18d310298599537de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c4b578f30e784ce18d310298599537de; no voters:
I20260620 09:26:22.083600 20168 leader_election.cc:288] T 00000000000000000000000000000000 P c4b578f30e784ce18d310298599537de [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:26:22.083629 20173 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c4b578f30e784ce18d310298599537de [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:26:22.083736 20173 raft_consensus.cc:686] T 00000000000000000000000000000000 P c4b578f30e784ce18d310298599537de [term 1 LEADER]: Becoming Leader. State: Replica: c4b578f30e784ce18d310298599537de, State: Running, Role: LEADER
I20260620 09:26:22.083813 20173 consensus_queue.cc:237] T 00000000000000000000000000000000 P c4b578f30e784ce18d310298599537de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4b578f30e784ce18d310298599537de" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 45569 } }
I20260620 09:26:22.083854 20168 sys_catalog.cc:564] T 00000000000000000000000000000000 P c4b578f30e784ce18d310298599537de [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:26:22.084157 20173 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4b578f30e784ce18d310298599537de [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c4b578f30e784ce18d310298599537de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4b578f30e784ce18d310298599537de" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 45569 } } }
I20260620 09:26:22.084228 20173 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4b578f30e784ce18d310298599537de [sys.catalog]: This master's current role is: LEADER
I20260620 09:26:22.084350 20173 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4b578f30e784ce18d310298599537de [sys.catalog]: SysCatalogTable state changed. Reason: New leader c4b578f30e784ce18d310298599537de. Latest consensus state: current_term: 1 leader_uuid: "c4b578f30e784ce18d310298599537de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4b578f30e784ce18d310298599537de" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 45569 } } }
I20260620 09:26:22.084492 20173 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4b578f30e784ce18d310298599537de [sys.catalog]: This master's current role is: LEADER
I20260620 09:26:22.084671 20178 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:26:22.085235 20178 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:26:22.088017 20178 catalog_manager.cc:1347] Generated new cluster ID: d3c4e107bc074a5587e22c0459277136
I20260620 09:26:22.088078 20178 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 09:26:22 dist-test-slave-706q krb5kdc[20087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947581, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.43.190@KRBTEST.COM
I20260620 09:26:22.092839 20172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:22.080859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45854 (local address 127.16.43.190:45569)
0620 09:26:22.081053 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:22.081061 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:22.081086 (+ 25us) server_negotiation.cc:407] Connection header received
0620 09:26:22.082307 (+ 1221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:22.082315 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:22.082373 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:22.082577 (+ 204us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:22.083344 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:22.085078 (+ 1734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:22.086333 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:22.086981 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:22.087107 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:22.089517 (+ 2410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:22.089522 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:22.089522 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:26:22.089540 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:22.091265 (+ 1725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:22.091709 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:22.091712 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:22.091712 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:22.091816 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:22.092030 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:22.092031 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:22.092032 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:22.092269 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:22.092439 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:22.092535 (+ 96us) server_negotiation.cc:299] Negotiation successful
0620 09:26:22.092676 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":101,"threads_started":1}
I20260620 09:26:22.095466 20178 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:26:22.096027 20178 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:26:22.098032 20178 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c4b578f30e784ce18d310298599537de: Generated new TSK 0
I20260620 09:26:22.098206 20178 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.43.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.43.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.43.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.129.keytab.
Entry for principal kudu/127.16.43.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.129.keytab.
I20260620 09:26:22.134644 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781947562949448-16558-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781947562949448-16558-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781947562949448-16558-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781947562949448-16558-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.43.129:0
--local_ip_for_outbound_sockets=127.16.43.129
--webserver_interface=127.16.43.129
--webserver_port=0
--tserver_master_addrs=127.16.43.190:45569
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.129.keytab
--principal=kudu/127.16.43.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.43.148:35451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781947562949448-16558-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781947562949448-16558-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781947562949448-16558-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:26:22.235076 20194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:26:22.235351 20194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:26:22.235378 20194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:26:22.236747 20194 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:26:22.236804 20194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:26:22.236833 20194 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.43.129
I20260620 09:26:22.238404 20194 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:35451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781947562949448-16558-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781947562949448-16558-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.129.keytab
--principal=kudu/127.16.43.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.43.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781947562949448-16558-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.43.129
--webserver_port=0
--tserver_master_addrs=127.16.43.190:45569
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20194
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.43.129
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781947562949448-16558-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:26:22.238658 20194 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:26:22.238924 20194 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:26:22.245155 20199 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:26:22.245186 20200 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:26:22.245500 20202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:26:22.245662 20194 server_base.cc:1034] running on GCE node
I20260620 09:26:22.245875 20194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:26:22.246156 20194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:26:22.247336 20194 hybrid_clock.cc:648] HybridClock initialized: now 1781947582247322 us; error 30 us; skew 500 ppm
Jun 20 09:26:22 dist-test-slave-706q krb5kdc[20087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947582, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:26:22.250265 20194 init.cc:377] Logged in from keytab as kudu/127.16.43.129@KRBTEST.COM (short username kudu)
I20260620 09:26:22.250869 20194 webserver.cc:460] Webserver started at http://127.16.43.129:45001/ using document root <none> and password file <none>
I20260620 09:26:22.251109 20194 fs_manager.cc:359] Metadata directory not provided
I20260620 09:26:22.251183 20194 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:26:22.251339 20194 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:26:22.252518 20194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781947562949448-16558-0/minicluster-data/ts-0/data/instance:
uuid: "f27916cd2eff4ce38befaed42295b32a"
format_stamp: "Formatted at 2026-06-20 09:26:22 on dist-test-slave-706q"
server_key: "3651b87b33bca2da62825f593fcf3a69"
server_key_iv: "c73b7e51ea2ccce26dd1d581828930c7"
server_key_version: "encryptionkey@0"
I20260620 09:26:22.252887 20194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781947562949448-16558-0/minicluster-data/ts-0/wal/instance:
uuid: "f27916cd2eff4ce38befaed42295b32a"
format_stamp: "Formatted at 2026-06-20 09:26:22 on dist-test-slave-706q"
server_key: "3651b87b33bca2da62825f593fcf3a69"
server_key_iv: "c73b7e51ea2ccce26dd1d581828930c7"
server_key_version: "encryptionkey@0"
I20260620 09:26:22.254263 20194 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 09:26:22.255092 20209 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:22.255303 20194 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 09:26:22.255471 20194 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781947562949448-16558-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781947562949448-16558-0/minicluster-data/ts-0/wal
uuid: "f27916cd2eff4ce38befaed42295b32a"
format_stamp: "Formatted at 2026-06-20 09:26:22 on dist-test-slave-706q"
server_key: "3651b87b33bca2da62825f593fcf3a69"
server_key_iv: "c73b7e51ea2ccce26dd1d581828930c7"
server_key_version: "encryptionkey@0"
I20260620 09:26:22.255610 20194 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781947562949448-16558-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781947562949448-16558-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781947562949448-16558-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:26:22.276412 20194 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:26:22.278560 20194 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:26:22.278743 20194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:26:22.279018 20194 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:26:22.279351 20194 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:26:22.279408 20194 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:22.279450 20194 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:26:22.279503 20194 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:22.286154 20194 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.129:44093
I20260620 09:26:22.286198 20322 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.129:44093 every 8 connection(s)
I20260620 09:26:22.286582 20194 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781947562949448-16558-0/minicluster-data/ts-0/data/info.pb
I20260620 09:26:22.290863 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 20194
I20260620 09:26:22.290992 16558 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781947562949448-16558-0/minicluster-data/ts-0/wal/instance
I20260620 09:26:22.291194 16558 external_mini_cluster.cc:1428] Setting key 1c7b9251199688f048a8757315e51043
Jun 20 09:26:22 dist-test-slave-706q krb5kdc[20087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947582, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.129@KRBTEST.COM for kudu/127.16.43.190@KRBTEST.COM
I20260620 09:26:22.297533 20172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:22.287328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.129:55169 (local address 127.16.43.190:45569)
0620 09:26:22.287395 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:22.287398 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:22.287729 (+ 331us) server_negotiation.cc:407] Connection header received
0620 09:26:22.288714 (+ 985us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:22.288715 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:22.288730 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:22.288780 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:22.289373 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:22.289873 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:22.291025 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:22.291343 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:22.291419 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:22.294176 (+ 2757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:22.294178 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:22.294178 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:26:22.294189 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:22.295922 (+ 1733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:22.296410 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:22.296411 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:22.296412 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:22.296485 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:22.296778 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:22.296779 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:22.296779 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:22.296916 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:22.297016 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:22.297321 (+ 305us) server_negotiation.cc:299] Negotiation successful
0620 09:26:22.297431 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 09:26:22.297755 20325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:22.287313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:45569 (local address 127.16.43.129:55169)
0620 09:26:22.287562 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:22.287641 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:22.288619 (+ 978us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:22.288843 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:22.288846 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:22.289098 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:22.289292 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:22.289296 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:22.289979 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:22.289980 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:22.290858 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:22.290870 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:22.291447 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:22.291448 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:22.291646 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:22.292342 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:26:22.292351 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:26:22.294053 (+ 1702us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:26:22.296031 (+ 1978us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:22.296040 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:22.296043 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:22.296304 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:22.296577 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:22.296578 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:22.296579 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:22.296702 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:22.297003 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:26:22.297011 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:26:22.297174 (+ 163us) client_negotiation.cc:769] Sending connection context
0620 09:26:22.297299 (+ 125us) client_negotiation.cc:240] Negotiation successful
0620 09:26:22.297477 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":97,"threads_started":1}
I20260620 09:26:22.298463 20323 heartbeater.cc:347] Connected to a master server at 127.16.43.190:45569
I20260620 09:26:22.298592 20323 heartbeater.cc:464] Registering TS with master...
I20260620 09:26:22.298789 20323 heartbeater.cc:511] Master 127.16.43.190:45569 requested a full tablet report, sending...
I20260620 09:26:22.299404 20132 ts_manager.cc:194] Registered new tserver with Master: f27916cd2eff4ce38befaed42295b32a (127.16.43.129:44093)
WARNING: no policy specified for kudu/127.16.43.130@KRBTEST.COM; defaulting to no policy
I20260620 09:26:22.300132 20132 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.43.129@KRBTEST.COM'} at 127.16.43.129:55169
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.43.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.43.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.130.keytab.
Entry for principal kudu/127.16.43.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.130.keytab.
I20260620 09:26:22.319121 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781947562949448-16558-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781947562949448-16558-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781947562949448-16558-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781947562949448-16558-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.43.130:0
--local_ip_for_outbound_sockets=127.16.43.130
--webserver_interface=127.16.43.130
--webserver_port=0
--tserver_master_addrs=127.16.43.190:45569
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.130.keytab
--principal=kudu/127.16.43.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.43.148:35451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781947562949448-16558-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781947562949448-16558-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781947562949448-16558-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:26:22.403682 20328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:26:22.403945 20328 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:26:22.403978 20328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:26:22.405390 20328 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:26:22.405529 20328 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:26:22.405566 20328 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.43.130
I20260620 09:26:22.407409 20328 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:35451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781947562949448-16558-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781947562949448-16558-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.130.keytab
--principal=kudu/127.16.43.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.43.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781947562949448-16558-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.43.130
--webserver_port=0
--tserver_master_addrs=127.16.43.190:45569
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20328
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.43.130
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781947562949448-16558-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:26:22.407752 20328 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:26:22.408023 20328 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:26:22.413506 20334 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:26:22.413529 20333 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:26:22.413522 20336 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:26:22.413729 20328 server_base.cc:1034] running on GCE node
I20260620 09:26:22.414079 20328 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:26:22.414647 20328 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:26:22.415807 20328 hybrid_clock.cc:648] HybridClock initialized: now 1781947582415780 us; error 43 us; skew 500 ppm
Jun 20 09:26:22 dist-test-slave-706q krb5kdc[20087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947582, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:26:22.418522 20328 init.cc:377] Logged in from keytab as kudu/127.16.43.130@KRBTEST.COM (short username kudu)
I20260620 09:26:22.418977 20328 webserver.cc:460] Webserver started at http://127.16.43.130:38755/ using document root <none> and password file <none>
I20260620 09:26:22.419226 20328 fs_manager.cc:359] Metadata directory not provided
I20260620 09:26:22.419297 20328 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:26:22.419441 20328 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:26:22.420598 20328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781947562949448-16558-0/minicluster-data/ts-1/data/instance:
uuid: "ce9acca8bc5541a2859d5f21f971c44a"
format_stamp: "Formatted at 2026-06-20 09:26:22 on dist-test-slave-706q"
server_key: "28017a2692037fd6de2821d115c2b442"
server_key_iv: "2e10a7663a7669e7eb7bb4fcef1cd292"
server_key_version: "encryptionkey@0"
I20260620 09:26:22.420960 20328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781947562949448-16558-0/minicluster-data/ts-1/wal/instance:
uuid: "ce9acca8bc5541a2859d5f21f971c44a"
format_stamp: "Formatted at 2026-06-20 09:26:22 on dist-test-slave-706q"
server_key: "28017a2692037fd6de2821d115c2b442"
server_key_iv: "2e10a7663a7669e7eb7bb4fcef1cd292"
server_key_version: "encryptionkey@0"
I20260620 09:26:22.422312 20328 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 09:26:22.423116 20343 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:22.423344 20328 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:26:22.423448 20328 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781947562949448-16558-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781947562949448-16558-0/minicluster-data/ts-1/wal
uuid: "ce9acca8bc5541a2859d5f21f971c44a"
format_stamp: "Formatted at 2026-06-20 09:26:22 on dist-test-slave-706q"
server_key: "28017a2692037fd6de2821d115c2b442"
server_key_iv: "2e10a7663a7669e7eb7bb4fcef1cd292"
server_key_version: "encryptionkey@0"
I20260620 09:26:22.423554 20328 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781947562949448-16558-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781947562949448-16558-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781947562949448-16558-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:26:22.438424 20328 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:26:22.440718 20328 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:26:22.440912 20328 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:26:22.441190 20328 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:26:22.441548 20328 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:26:22.441608 20328 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:22.441656 20328 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:26:22.441705 20328 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:22.448720 20328 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.130:37475
I20260620 09:26:22.448777 20456 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.130:37475 every 8 connection(s)
I20260620 09:26:22.449144 20328 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781947562949448-16558-0/minicluster-data/ts-1/data/info.pb
I20260620 09:26:22.454643 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 20328
I20260620 09:26:22.454764 16558 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781947562949448-16558-0/minicluster-data/ts-1/wal/instance
I20260620 09:26:22.455061 16558 external_mini_cluster.cc:1428] Setting key 022b500cb82955fcf4020bfb3fe89e68
Jun 20 09:26:22 dist-test-slave-706q krb5kdc[20087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947582, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.130@KRBTEST.COM for kudu/127.16.43.190@KRBTEST.COM
I20260620 09:26:22.460012 20172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:22.449987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.130:43849 (local address 127.16.43.190:45569)
0620 09:26:22.450067 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:22.450069 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:22.450263 (+ 194us) server_negotiation.cc:407] Connection header received
0620 09:26:22.451187 (+ 924us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:22.451188 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:22.451198 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:22.451251 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:22.451907 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:22.452419 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:22.453454 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:22.453649 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:22.453707 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:22.456239 (+ 2532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:22.456240 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:22.456241 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:26:22.456248 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:22.458212 (+ 1964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:22.458790 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:22.458792 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:22.458792 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:22.458861 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:22.459116 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:22.459118 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:22.459118 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:22.459244 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:22.459339 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:22.459793 (+ 454us) server_negotiation.cc:299] Negotiation successful
0620 09:26:22.459888 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 09:26:22.460188 20459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:22.449946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:45569 (local address 127.16.43.130:43849)
0620 09:26:22.450133 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:22.450170 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:22.451072 (+ 902us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:22.451333 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:22.451336 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:22.451620 (+ 284us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:22.451822 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:22.451827 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:22.452530 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:22.452531 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:22.453371 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:22.453374 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:22.453727 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:22.453728 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:22.453830 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:22.454438 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:26:22.454447 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:26:22.456147 (+ 1700us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:26:22.458332 (+ 2185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:22.458339 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:22.458340 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:22.458705 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:22.458934 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:22.458935 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:22.458935 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:22.459038 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:22.459373 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:26:22.459379 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:26:22.459637 (+ 258us) client_negotiation.cc:769] Sending connection context
0620 09:26:22.459759 (+ 122us) client_negotiation.cc:240] Negotiation successful
0620 09:26:22.459930 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":103,"threads_started":1}
I20260620 09:26:22.460700 20457 heartbeater.cc:347] Connected to a master server at 127.16.43.190:45569
I20260620 09:26:22.460807 20457 heartbeater.cc:464] Registering TS with master...
I20260620 09:26:22.461010 20457 heartbeater.cc:511] Master 127.16.43.190:45569 requested a full tablet report, sending...
I20260620 09:26:22.461428 20132 ts_manager.cc:194] Registered new tserver with Master: ce9acca8bc5541a2859d5f21f971c44a (127.16.43.130:37475)
I20260620 09:26:22.461823 20132 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.43.130@KRBTEST.COM'} at 127.16.43.130:43849
WARNING: no policy specified for kudu/127.16.43.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.43.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.43.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.131.keytab.
Entry for principal kudu/127.16.43.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.131.keytab.
I20260620 09:26:22.482913 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781947562949448-16558-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781947562949448-16558-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781947562949448-16558-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781947562949448-16558-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.43.131:0
--local_ip_for_outbound_sockets=127.16.43.131
--webserver_interface=127.16.43.131
--webserver_port=0
--tserver_master_addrs=127.16.43.190:45569
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.131.keytab
--principal=kudu/127.16.43.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.43.148:35451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781947562949448-16558-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781947562949448-16558-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781947562949448-16558-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:26:22.565263 20462 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:26:22.565485 20462 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:26:22.565510 20462 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:26:22.566607 20462 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:26:22.566641 20462 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:26:22.566668 20462 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.43.131
I20260620 09:26:22.568202 20462 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:35451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781947562949448-16558-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781947562949448-16558-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.131.keytab
--principal=kudu/127.16.43.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.43.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781947562949448-16558-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.43.131
--webserver_port=0
--tserver_master_addrs=127.16.43.190:45569
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20462
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.43.131
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781947562949448-16558-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:26:22.568396 20462 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:26:22.568586 20462 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:26:22.573879 20467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:26:22.573988 20462 server_base.cc:1034] running on GCE node
W20260620 09:26:22.573899 20468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:26:22.573907 20470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:26:22.574379 20462 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:26:22.574589 20462 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:26:22.575724 20462 hybrid_clock.cc:648] HybridClock initialized: now 1781947582575706 us; error 40 us; skew 500 ppm
Jun 20 09:26:22 dist-test-slave-706q krb5kdc[20087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947582, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:26:22.578245 20462 init.cc:377] Logged in from keytab as kudu/127.16.43.131@KRBTEST.COM (short username kudu)
I20260620 09:26:22.578747 20462 webserver.cc:460] Webserver started at http://127.16.43.131:44595/ using document root <none> and password file <none>
I20260620 09:26:22.578976 20462 fs_manager.cc:359] Metadata directory not provided
I20260620 09:26:22.579052 20462 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:26:22.579200 20462 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:26:22.580334 20462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781947562949448-16558-0/minicluster-data/ts-2/data/instance:
uuid: "0b9cdecfc613435e98f92d1876bc0c95"
format_stamp: "Formatted at 2026-06-20 09:26:22 on dist-test-slave-706q"
server_key: "764d76fe7c6754dbf94b73c78a3f85be"
server_key_iv: "6f824db2d29af4a3dd9432e6c3c722e7"
server_key_version: "encryptionkey@0"
I20260620 09:26:22.580693 20462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781947562949448-16558-0/minicluster-data/ts-2/wal/instance:
uuid: "0b9cdecfc613435e98f92d1876bc0c95"
format_stamp: "Formatted at 2026-06-20 09:26:22 on dist-test-slave-706q"
server_key: "764d76fe7c6754dbf94b73c78a3f85be"
server_key_iv: "6f824db2d29af4a3dd9432e6c3c722e7"
server_key_version: "encryptionkey@0"
I20260620 09:26:22.582131 20462 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 09:26:22.582934 20477 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:22.583214 20462 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 09:26:22.583352 20462 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781947562949448-16558-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781947562949448-16558-0/minicluster-data/ts-2/wal
uuid: "0b9cdecfc613435e98f92d1876bc0c95"
format_stamp: "Formatted at 2026-06-20 09:26:22 on dist-test-slave-706q"
server_key: "764d76fe7c6754dbf94b73c78a3f85be"
server_key_iv: "6f824db2d29af4a3dd9432e6c3c722e7"
server_key_version: "encryptionkey@0"
I20260620 09:26:22.583546 20462 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781947562949448-16558-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781947562949448-16558-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781947562949448-16558-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:26:22.602264 20462 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:26:22.604488 20462 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:26:22.604712 20462 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:26:22.604967 20462 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:26:22.605321 20462 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:26:22.605381 20462 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:22.605422 20462 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:26:22.605475 20462 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:22.611444 20462 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.131:43581
I20260620 09:26:22.611498 20590 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.131:43581 every 8 connection(s)
I20260620 09:26:22.611884 20462 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781947562949448-16558-0/minicluster-data/ts-2/data/info.pb
Jun 20 09:26:22 dist-test-slave-706q krb5kdc[20087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947582, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.131@KRBTEST.COM for kudu/127.16.43.190@KRBTEST.COM
I20260620 09:26:22.618383 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 20462
I20260620 09:26:22.618494 16558 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781947562949448-16558-0/minicluster-data/ts-2/wal/instance
I20260620 09:26:22.618673 16558 external_mini_cluster.cc:1428] Setting key 5c675cd4564d7ef1d36159eda015af94
I20260620 09:26:22.622066 20172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:22.612925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.131:56999 (local address 127.16.43.190:45569)
0620 09:26:22.612971 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:22.612973 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:22.613161 (+ 188us) server_negotiation.cc:407] Connection header received
0620 09:26:22.614133 (+ 972us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:22.614134 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:22.614146 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:22.614210 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:22.614817 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:22.615332 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:22.616407 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:22.616620 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:22.616687 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:22.618945 (+ 2258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:22.618947 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:22.618947 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:26:22.618955 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:22.620598 (+ 1643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:22.621048 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:22.621049 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:22.621049 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:22.621110 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:22.621379 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:22.621380 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:22.621380 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:22.621500 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:22.621581 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:22.621895 (+ 314us) server_negotiation.cc:299] Negotiation successful
0620 09:26:22.621979 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 09:26:22.622255 20593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:22.612905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:45569 (local address 127.16.43.131:56999)
0620 09:26:22.613050 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:22.613086 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:22.614014 (+ 928us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:22.614270 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:22.614274 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:22.614520 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:22.614721 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:22.614725 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:22.615412 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:22.615413 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:22.616310 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:22.616312 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:22.616705 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:22.616706 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:22.616808 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:22.617285 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:26:22.617293 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:26:22.618843 (+ 1550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:26:22.620704 (+ 1861us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:22.620711 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:22.620714 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:22.620957 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:22.621177 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:22.621177 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:22.621178 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:22.621302 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:22.621570 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:26:22.621576 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:26:22.621725 (+ 149us) client_negotiation.cc:769] Sending connection context
0620 09:26:22.621860 (+ 135us) client_negotiation.cc:240] Negotiation successful
0620 09:26:22.622014 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":81,"threads_started":1}
I20260620 09:26:22.622834 20591 heartbeater.cc:347] Connected to a master server at 127.16.43.190:45569
I20260620 09:26:22.622920 20591 heartbeater.cc:464] Registering TS with master...
I20260620 09:26:22.623121 20591 heartbeater.cc:511] Master 127.16.43.190:45569 requested a full tablet report, sending...
I20260620 09:26:22.623700 20132 ts_manager.cc:194] Registered new tserver with Master: 0b9cdecfc613435e98f92d1876bc0c95 (127.16.43.131:43581)
I20260620 09:26:22.624104 20132 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.43.131@KRBTEST.COM'} at 127.16.43.131:56999
I20260620 09:26:22.630635 16558 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 09:26:22 dist-test-slave-706q krb5kdc[20087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947582, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 09:26:22 dist-test-slave-706q krb5kdc[20087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947582, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.43.129@KRBTEST.COM
I20260620 09:26:22.654723 20598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:22.646637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42870 (local address 127.16.43.129:44093)
0620 09:26:22.646798 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:22.646808 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:22.646819 (+ 11us) server_negotiation.cc:407] Connection header received
0620 09:26:22.646847 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:22.646849 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:22.646863 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:22.646939 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:22.647543 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:22.648306 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:22.649077 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:22.649387 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:22.649442 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:22.651892 (+ 2450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:22.651900 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:22.651900 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:26:22.651907 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:22.653372 (+ 1465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:22.653864 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:22.653866 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:22.653867 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:22.653920 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:22.654159 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:22.654159 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:22.654160 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:22.654288 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:22.654393 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:22.654579 (+ 186us) server_negotiation.cc:299] Negotiation successful
0620 09:26:22.654642 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":89,"threads_started":1}
Jun 20 09:26:22 dist-test-slave-706q krb5kdc[20087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947582, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.43.190@KRBTEST.COM
I20260620 09:26:22.664113 20172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:22.656192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45868 (local address 127.16.43.190:45569)
0620 09:26:22.656247 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:22.656249 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:22.656257 (+ 8us) server_negotiation.cc:407] Connection header received
0620 09:26:22.656335 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:22.656336 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:22.656344 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:22.656393 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:22.656936 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:22.657421 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:22.658246 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:22.658531 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:22.658604 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:22.661438 (+ 2834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:22.661439 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:22.661440 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:26:22.661447 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:22.662829 (+ 1382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:22.663169 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:22.663170 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:22.663170 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:22.663218 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:22.663429 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:22.663430 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:22.663430 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:22.663686 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:22.663768 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:22.663970 (+ 202us) server_negotiation.cc:299] Negotiation successful
0620 09:26:22.664045 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
W20260620 09:26:22.664402 20132 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45868
I20260620 09:26:22.664827 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 20194
I20260620 09:26:22.670050 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 20328
I20260620 09:26:22.675256 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 20462
I20260620 09:26:22.679915 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 20100
2026-06-20T09:26:22Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1940 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781947562949448-16558-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:26:22 dist-test-slave-706q krb5kdc[20603](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:26:22 dist-test-slave-706q krb5kdc[20603](info): set up 2 sockets
Jun 20 09:26:22 dist-test-slave-706q krb5kdc[20603](info): commencing operation
krb5kdc: starting...
W20260620 09:26:23.725406 16558 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781947562949448-16558-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781947562949448-16558-0/krb5kdc/test-user.keytab.
Jun 20 09:26:23 dist-test-slave-706q krb5kdc[20603](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947583, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:26:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:26:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.43.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.43.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.43.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.190.keytab.
Entry for principal kudu/127.16.43.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.190.keytab.
I20260620 09:26:23.850945 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781947562949448-16558-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781947562949448-16558-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781947562949448-16558-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.43.190:35973
--webserver_interface=127.16.43.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.190.keytab
--principal=kudu/127.16.43.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.43.148:37383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.43.190:35973
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781947562949448-16558-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781947562949448-16558-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781947562949448-16558-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:26:23.932132 20616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:26:23.932346 20616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:26:23.932370 20616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:26:23.933583 20616 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:26:23.933624 20616 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:26:23.933640 20616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:26:23.933653 20616 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:26:23.933665 20616 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:26:23.935204 20616 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:37383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781947562949448-16558-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781947562949448-16558-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.43.190:35973
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.190.keytab
--principal=kudu/127.16.43.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.43.190:35973
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.43.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20616
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781947562949448-16558-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:26:23.935478 20616 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:26:23.935743 20616 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:26:23.940999 20624 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:26:23.941062 20622 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:26:23.940961 20621 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:26:23.940934 20616 server_base.cc:1034] running on GCE node
I20260620 09:26:23.941567 20616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:26:23.941890 20616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:26:23.943086 20616 hybrid_clock.cc:648] HybridClock initialized: now 1781947583943049 us; error 52 us; skew 500 ppm
Jun 20 09:26:23 dist-test-slave-706q krb5kdc[20603](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947583, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:26:23.946130 20616 init.cc:377] Logged in from keytab as kudu/127.16.43.190@KRBTEST.COM (short username kudu)
I20260620 09:26:23.946583 20616 fs_manager.cc:359] Metadata directory not provided
I20260620 09:26:23.946672 20616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:26:23.946841 20616 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:26:23.948189 20616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781947562949448-16558-0/minicluster-data/master-0/data/instance:
uuid: "d3d9bbbc9a084b8a9b9ba26971b72e0e"
format_stamp: "Formatted at 2026-06-20 09:26:23 on dist-test-slave-706q"
server_key: "2cb77287df261e61d1cab7f1e3cc4f32"
server_key_iv: "ceb7dfddf634d5c7f17406674158cb60"
server_key_version: "encryptionkey@0"
I20260620 09:26:23.948621 20616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781947562949448-16558-0/minicluster-data/master-0/wal/instance:
uuid: "d3d9bbbc9a084b8a9b9ba26971b72e0e"
format_stamp: "Formatted at 2026-06-20 09:26:23 on dist-test-slave-706q"
server_key: "2cb77287df261e61d1cab7f1e3cc4f32"
server_key_iv: "ceb7dfddf634d5c7f17406674158cb60"
server_key_version: "encryptionkey@0"
I20260620 09:26:23.950095 20616 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 09:26:23.951004 20630 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:23.951211 20616 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:26:23.951320 20616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781947562949448-16558-0/minicluster-data/master-0/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781947562949448-16558-0/minicluster-data/master-0/wal
uuid: "d3d9bbbc9a084b8a9b9ba26971b72e0e"
format_stamp: "Formatted at 2026-06-20 09:26:23 on dist-test-slave-706q"
server_key: "2cb77287df261e61d1cab7f1e3cc4f32"
server_key_iv: "ceb7dfddf634d5c7f17406674158cb60"
server_key_version: "encryptionkey@0"
I20260620 09:26:23.951396 20616 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781947562949448-16558-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781947562949448-16558-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781947562949448-16558-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:26:23.972760 20616 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:26:23.975179 20616 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:26:23.975353 20616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:26:23.979496 20616 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.190:35973
I20260620 09:26:23.979658 20682 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.190:35973 every 8 connection(s)
I20260620 09:26:23.979925 20616 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
I20260620 09:26:23.980911 20683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:26:23.983148 20683 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3d9bbbc9a084b8a9b9ba26971b72e0e: Bootstrap starting.
I20260620 09:26:23.983731 20683 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d3d9bbbc9a084b8a9b9ba26971b72e0e: Neither blocks nor log segments found. Creating new log.
I20260620 09:26:23.983968 20683 log.cc:824] T 00000000000000000000000000000000 P d3d9bbbc9a084b8a9b9ba26971b72e0e: Log is configured to *not* fsync() on all Append() calls
I20260620 09:26:23.984663 20683 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3d9bbbc9a084b8a9b9ba26971b72e0e: No bootstrap required, opened a new log
I20260620 09:26:23.986219 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 20616
I20260620 09:26:23.986229 20683 raft_consensus.cc:348] T 00000000000000000000000000000000 P d3d9bbbc9a084b8a9b9ba26971b72e0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3d9bbbc9a084b8a9b9ba26971b72e0e" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 35973 } }
I20260620 09:26:23.986361 16558 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781947562949448-16558-0/minicluster-data/master-0/wal/instance
I20260620 09:26:23.986410 20683 raft_consensus.cc:374] T 00000000000000000000000000000000 P d3d9bbbc9a084b8a9b9ba26971b72e0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:26:23.986447 20683 raft_consensus.cc:729] T 00000000000000000000000000000000 P d3d9bbbc9a084b8a9b9ba26971b72e0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3d9bbbc9a084b8a9b9ba26971b72e0e, State: Initialized, Role: FOLLOWER
I20260620 09:26:23.986599 16558 external_mini_cluster.cc:1428] Setting key 069d58adf50c344bfbe09ddbc9e66518
I20260620 09:26:23.986603 20683 consensus_queue.cc:260] T 00000000000000000000000000000000 P d3d9bbbc9a084b8a9b9ba26971b72e0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3d9bbbc9a084b8a9b9ba26971b72e0e" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 35973 } }
I20260620 09:26:23.986703 20683 raft_consensus.cc:388] T 00000000000000000000000000000000 P d3d9bbbc9a084b8a9b9ba26971b72e0e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:26:23.986744 20683 raft_consensus.cc:482] T 00000000000000000000000000000000 P d3d9bbbc9a084b8a9b9ba26971b72e0e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:26:23.986790 20683 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d3d9bbbc9a084b8a9b9ba26971b72e0e [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:26:23.987445 20683 raft_consensus.cc:504] T 00000000000000000000000000000000 P d3d9bbbc9a084b8a9b9ba26971b72e0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3d9bbbc9a084b8a9b9ba26971b72e0e" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 35973 } }
I20260620 09:26:23.987591 20683 leader_election.cc:302] T 00000000000000000000000000000000 P d3d9bbbc9a084b8a9b9ba26971b72e0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d3d9bbbc9a084b8a9b9ba26971b72e0e; no voters:
I20260620 09:26:23.987782 20683 leader_election.cc:288] T 00000000000000000000000000000000 P d3d9bbbc9a084b8a9b9ba26971b72e0e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:26:23.987816 20687 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d3d9bbbc9a084b8a9b9ba26971b72e0e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:26:23.987924 20687 raft_consensus.cc:686] T 00000000000000000000000000000000 P d3d9bbbc9a084b8a9b9ba26971b72e0e [term 1 LEADER]: Becoming Leader. State: Replica: d3d9bbbc9a084b8a9b9ba26971b72e0e, State: Running, Role: LEADER
I20260620 09:26:23.988051 20683 sys_catalog.cc:564] T 00000000000000000000000000000000 P d3d9bbbc9a084b8a9b9ba26971b72e0e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:26:23.988070 20687 consensus_queue.cc:237] T 00000000000000000000000000000000 P d3d9bbbc9a084b8a9b9ba26971b72e0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3d9bbbc9a084b8a9b9ba26971b72e0e" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 35973 } }
I20260620 09:26:23.988409 20687 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3d9bbbc9a084b8a9b9ba26971b72e0e [sys.catalog]: SysCatalogTable state changed. Reason: New leader d3d9bbbc9a084b8a9b9ba26971b72e0e. Latest consensus state: current_term: 1 leader_uuid: "d3d9bbbc9a084b8a9b9ba26971b72e0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3d9bbbc9a084b8a9b9ba26971b72e0e" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 35973 } } }
I20260620 09:26:23.988430 20689 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3d9bbbc9a084b8a9b9ba26971b72e0e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d3d9bbbc9a084b8a9b9ba26971b72e0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3d9bbbc9a084b8a9b9ba26971b72e0e" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 35973 } } }
I20260620 09:26:23.988576 20687 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3d9bbbc9a084b8a9b9ba26971b72e0e [sys.catalog]: This master's current role is: LEADER
I20260620 09:26:23.988672 20689 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3d9bbbc9a084b8a9b9ba26971b72e0e [sys.catalog]: This master's current role is: LEADER
W20260620 09:26:23.989933 20704 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d3d9bbbc9a084b8a9b9ba26971b72e0e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 09:26:23.989997 20704 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 09:26:23.990058 20697 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:26:23.990259 20697 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:26:23.992223 20697 catalog_manager.cc:1347] Generated new cluster ID: 3ad9b94d54af487fa1366f7240a9cf21
I20260620 09:26:23.992282 20697 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 09:26:23 dist-test-slave-706q krb5kdc[20603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947583, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.43.190@KRBTEST.COM
I20260620 09:26:24.000020 20688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:23.987164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41644 (local address 127.16.43.190:35973)
0620 09:26:23.988021 (+ 857us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:23.988032 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:23.988074 (+ 42us) server_negotiation.cc:407] Connection header received
0620 09:26:23.988221 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:23.988228 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:23.988288 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:23.989327 (+ 1039us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:23.989447 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:23.990976 (+ 1529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:23.992168 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:23.992862 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:23.992996 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:23.995923 (+ 2927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:23.995928 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:23.995928 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:26:23.995941 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:23.998057 (+ 2116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:23.998509 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:23.998513 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:23.998513 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:23.998628 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:23.998847 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:23.998848 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:23.998849 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:23.999177 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:23.999351 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:23.999633 (+ 282us) server_negotiation.cc:299] Negotiation successful
0620 09:26:23.999787 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":807,"thread_start_us":665,"threads_started":1}
I20260620 09:26:24.000284 20697 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:26:24.000996 20697 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:26:24.005792 20697 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d3d9bbbc9a084b8a9b9ba26971b72e0e: Generated new TSK 0
I20260620 09:26:24.006011 20697 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.43.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.43.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.43.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.129.keytab.
Entry for principal kudu/127.16.43.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.129.keytab.
I20260620 09:26:24.039376 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781947562949448-16558-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781947562949448-16558-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781947562949448-16558-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781947562949448-16558-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.43.129:0
--local_ip_for_outbound_sockets=127.16.43.129
--webserver_interface=127.16.43.129
--webserver_port=0
--tserver_master_addrs=127.16.43.190:35973
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.129.keytab
--principal=kudu/127.16.43.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.43.148:37383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781947562949448-16558-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781947562949448-16558-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781947562949448-16558-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:26:24.126466 20709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:26:24.126760 20709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:26:24.126806 20709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:26:24.128224 20709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:26:24.128297 20709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:26:24.128341 20709 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.43.129
I20260620 09:26:24.130105 20709 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:37383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781947562949448-16558-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781947562949448-16558-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.129.keytab
--principal=kudu/127.16.43.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.43.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781947562949448-16558-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.43.129
--webserver_port=0
--tserver_master_addrs=127.16.43.190:35973
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20709
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.43.129
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781947562949448-16558-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:26:24.130405 20709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:26:24.130659 20709 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:26:24.136018 20715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:26:24.136098 20714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:26:24.136189 20709 server_base.cc:1034] running on GCE node
W20260620 09:26:24.136018 20717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:26:24.136507 20709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:26:24.136788 20709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:26:24.137990 20709 hybrid_clock.cc:648] HybridClock initialized: now 1781947584137970 us; error 41 us; skew 500 ppm
Jun 20 09:26:24 dist-test-slave-706q krb5kdc[20603](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947584, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:26:24.140718 20709 init.cc:377] Logged in from keytab as kudu/127.16.43.129@KRBTEST.COM (short username kudu)
I20260620 09:26:24.141105 20709 fs_manager.cc:359] Metadata directory not provided
I20260620 09:26:24.141185 20709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:26:24.141336 20709 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:26:24.142438 20709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781947562949448-16558-0/minicluster-data/ts-0/data/instance:
uuid: "dab78762a30e450fab9339720be18fe3"
format_stamp: "Formatted at 2026-06-20 09:26:24 on dist-test-slave-706q"
server_key: "dfbb9f156dbf880edb7bdfeff87c403d"
server_key_iv: "cb9e47cd6e9f87853aca60317ed20f75"
server_key_version: "encryptionkey@0"
I20260620 09:26:24.142817 20709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781947562949448-16558-0/minicluster-data/ts-0/wal/instance:
uuid: "dab78762a30e450fab9339720be18fe3"
format_stamp: "Formatted at 2026-06-20 09:26:24 on dist-test-slave-706q"
server_key: "dfbb9f156dbf880edb7bdfeff87c403d"
server_key_iv: "cb9e47cd6e9f87853aca60317ed20f75"
server_key_version: "encryptionkey@0"
I20260620 09:26:24.144268 20709 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 09:26:24.145169 20723 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:24.145458 20709 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:26:24.145643 20709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781947562949448-16558-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781947562949448-16558-0/minicluster-data/ts-0/wal
uuid: "dab78762a30e450fab9339720be18fe3"
format_stamp: "Formatted at 2026-06-20 09:26:24 on dist-test-slave-706q"
server_key: "dfbb9f156dbf880edb7bdfeff87c403d"
server_key_iv: "cb9e47cd6e9f87853aca60317ed20f75"
server_key_version: "encryptionkey@0"
I20260620 09:26:24.145744 20709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781947562949448-16558-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781947562949448-16558-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781947562949448-16558-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:26:24.155764 20709 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:26:24.157801 20709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:26:24.157974 20709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:26:24.158243 20709 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:26:24.158617 20709 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:26:24.158664 20709 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:24.158700 20709 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:26:24.158727 20709 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:24.164906 20709 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.129:33733
I20260620 09:26:24.164944 20836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.129:33733 every 8 connection(s)
I20260620 09:26:24.165257 20709 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781947562949448-16558-0/minicluster-data/ts-0/data/info.pb
Jun 20 09:26:24 dist-test-slave-706q krb5kdc[20603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947584, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.129@KRBTEST.COM for kudu/127.16.43.190@KRBTEST.COM
I20260620 09:26:24.175158 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 20709
I20260620 09:26:24.175262 20688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:24.166057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.129:34331 (local address 127.16.43.190:35973)
0620 09:26:24.166143 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:24.166146 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:24.166369 (+ 223us) server_negotiation.cc:407] Connection header received
0620 09:26:24.166873 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:24.166875 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:24.166889 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:24.166967 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:24.167938 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:24.168511 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:24.169536 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:24.169755 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:24.169823 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:24.172128 (+ 2305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:24.172130 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:24.172130 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:26:24.172140 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:24.173725 (+ 1585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:24.174143 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:24.174145 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:24.174145 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:24.174201 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:24.174540 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:24.174541 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:24.174542 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:24.174670 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:24.174737 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:24.175047 (+ 310us) server_negotiation.cc:299] Negotiation successful
0620 09:26:24.175147 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 09:26:24.175277 16558 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781947562949448-16558-0/minicluster-data/ts-0/wal/instance
I20260620 09:26:24.175355 20839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:24.165985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:35973 (local address 127.16.43.129:34331)
0620 09:26:24.166209 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:24.166252 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:24.166747 (+ 495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:24.167015 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:24.167019 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:24.167281 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:24.167820 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:24.167825 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:24.168628 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:24.168629 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:24.169432 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:24.169434 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:24.169843 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:24.169844 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:24.169947 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:24.170523 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:26:24.170532 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:26:24.172025 (+ 1493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:26:24.173817 (+ 1792us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:24.173824 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:24.173825 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:24.174072 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:24.174285 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:24.174286 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:24.174286 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:24.174412 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:24.174744 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:26:24.174750 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:26:24.174882 (+ 132us) client_negotiation.cc:769] Sending connection context
0620 09:26:24.174981 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 09:26:24.175114 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":109,"threads_started":1}
I20260620 09:26:24.175472 16558 external_mini_cluster.cc:1428] Setting key f591b53f4795a224f151f5c5d2566a17
I20260620 09:26:24.176030 20837 heartbeater.cc:347] Connected to a master server at 127.16.43.190:35973
I20260620 09:26:24.176128 20837 heartbeater.cc:464] Registering TS with master...
I20260620 09:26:24.176326 20837 heartbeater.cc:511] Master 127.16.43.190:35973 requested a full tablet report, sending...
I20260620 09:26:24.176925 20646 ts_manager.cc:194] Registered new tserver with Master: dab78762a30e450fab9339720be18fe3 (127.16.43.129:33733)
I20260620 09:26:24.177856 20646 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.43.129@KRBTEST.COM'} at 127.16.43.129:34331
WARNING: no policy specified for kudu/127.16.43.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.43.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.43.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.130.keytab.
Entry for principal kudu/127.16.43.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.130.keytab.
I20260620 09:26:24.203090 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781947562949448-16558-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781947562949448-16558-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781947562949448-16558-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781947562949448-16558-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.43.130:0
--local_ip_for_outbound_sockets=127.16.43.130
--webserver_interface=127.16.43.130
--webserver_port=0
--tserver_master_addrs=127.16.43.190:35973
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.130.keytab
--principal=kudu/127.16.43.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.43.148:37383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781947562949448-16558-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781947562949448-16558-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781947562949448-16558-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:26:24.285017 20842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:26:24.285305 20842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:26:24.285348 20842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:26:24.286973 20842 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:26:24.287071 20842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:26:24.287125 20842 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.43.130
I20260620 09:26:24.289104 20842 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:37383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781947562949448-16558-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781947562949448-16558-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.130.keytab
--principal=kudu/127.16.43.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.43.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781947562949448-16558-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.43.130
--webserver_port=0
--tserver_master_addrs=127.16.43.190:35973
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20842
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.43.130
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781947562949448-16558-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:26:24.289379 20842 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:26:24.289655 20842 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 09:26:24.295018 20842 server_base.cc:1034] running on GCE node
W20260620 09:26:24.294977 20847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:26:24.294996 20848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:26:24.294997 20850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:26:24.295372 20842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:26:24.295723 20842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:26:24.296926 20842 hybrid_clock.cc:648] HybridClock initialized: now 1781947584296908 us; error 33 us; skew 500 ppm
Jun 20 09:26:24 dist-test-slave-706q krb5kdc[20603](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947584, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:26:24.299672 20842 init.cc:377] Logged in from keytab as kudu/127.16.43.130@KRBTEST.COM (short username kudu)
I20260620 09:26:24.300091 20842 fs_manager.cc:359] Metadata directory not provided
I20260620 09:26:24.300158 20842 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:26:24.300292 20842 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:26:24.301450 20842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781947562949448-16558-0/minicluster-data/ts-1/data/instance:
uuid: "6f161ceded5e47f7bf91e999713bc016"
format_stamp: "Formatted at 2026-06-20 09:26:24 on dist-test-slave-706q"
server_key: "87499dd5b5c459b304ea341fdd98f651"
server_key_iv: "791ed7d92c2995464d539b46dfbb1ca6"
server_key_version: "encryptionkey@0"
I20260620 09:26:24.301824 20842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781947562949448-16558-0/minicluster-data/ts-1/wal/instance:
uuid: "6f161ceded5e47f7bf91e999713bc016"
format_stamp: "Formatted at 2026-06-20 09:26:24 on dist-test-slave-706q"
server_key: "87499dd5b5c459b304ea341fdd98f651"
server_key_iv: "791ed7d92c2995464d539b46dfbb1ca6"
server_key_version: "encryptionkey@0"
I20260620 09:26:24.303143 20842 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 09:26:24.303929 20856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:24.304114 20842 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:24.304217 20842 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781947562949448-16558-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781947562949448-16558-0/minicluster-data/ts-1/wal
uuid: "6f161ceded5e47f7bf91e999713bc016"
format_stamp: "Formatted at 2026-06-20 09:26:24 on dist-test-slave-706q"
server_key: "87499dd5b5c459b304ea341fdd98f651"
server_key_iv: "791ed7d92c2995464d539b46dfbb1ca6"
server_key_version: "encryptionkey@0"
I20260620 09:26:24.304297 20842 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781947562949448-16558-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781947562949448-16558-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781947562949448-16558-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:26:24.314674 20842 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:26:24.316789 20842 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:26:24.316960 20842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:26:24.317214 20842 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:26:24.317538 20842 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:26:24.317590 20842 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:24.317626 20842 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:26:24.317655 20842 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:24.323483 20842 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.130:40553
I20260620 09:26:24.323542 20969 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.130:40553 every 8 connection(s)
I20260620 09:26:24.323805 20842 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781947562949448-16558-0/minicluster-data/ts-1/data/info.pb
I20260620 09:26:24.328378 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 20842
I20260620 09:26:24.328522 16558 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781947562949448-16558-0/minicluster-data/ts-1/wal/instance
I20260620 09:26:24.328733 16558 external_mini_cluster.cc:1428] Setting key ad63b7ff9fee73992ec01e35f7b2dc7b
Jun 20 09:26:24 dist-test-slave-706q krb5kdc[20603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947584, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.130@KRBTEST.COM for kudu/127.16.43.190@KRBTEST.COM
I20260620 09:26:24.334194 20688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:24.324539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.130:54327 (local address 127.16.43.190:35973)
0620 09:26:24.324606 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:24.324608 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:24.324813 (+ 205us) server_negotiation.cc:407] Connection header received
0620 09:26:24.325291 (+ 478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:24.325292 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:24.325304 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:24.325372 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:24.326352 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:24.326855 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:24.327933 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:24.328210 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:24.328291 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:24.330668 (+ 2377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:24.330670 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:24.330670 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:26:24.330679 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:24.332518 (+ 1839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:24.332995 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:24.332996 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:24.332997 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:24.333068 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:24.333340 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:24.333341 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:24.333342 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:24.333486 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:24.333699 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:24.333992 (+ 293us) server_negotiation.cc:299] Negotiation successful
0620 09:26:24.334103 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 09:26:24.334381 20972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:24.324489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:35973 (local address 127.16.43.130:54327)
0620 09:26:24.324672 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:24.324715 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:24.325187 (+ 472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:24.325448 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:24.325452 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:24.325752 (+ 300us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:24.326239 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:24.326244 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:24.326943 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:24.326944 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:24.327841 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:24.327843 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:24.328296 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:24.328297 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:24.328392 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:24.328939 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:26:24.328948 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:26:24.330565 (+ 1617us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:26:24.332632 (+ 2067us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:24.332640 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:24.332643 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:24.332885 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:24.333144 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:24.333145 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:24.333145 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:24.333263 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:24.333702 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:26:24.333710 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:26:24.333849 (+ 139us) client_negotiation.cc:769] Sending connection context
0620 09:26:24.333969 (+ 120us) client_negotiation.cc:240] Negotiation successful
0620 09:26:24.334135 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":91,"threads_started":1}
I20260620 09:26:24.334976 20970 heartbeater.cc:347] Connected to a master server at 127.16.43.190:35973
I20260620 09:26:24.335101 20970 heartbeater.cc:464] Registering TS with master...
I20260620 09:26:24.335278 20970 heartbeater.cc:511] Master 127.16.43.190:35973 requested a full tablet report, sending...
I20260620 09:26:24.335743 20646 ts_manager.cc:194] Registered new tserver with Master: 6f161ceded5e47f7bf91e999713bc016 (127.16.43.130:40553)
I20260620 09:26:24.336148 20646 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.43.130@KRBTEST.COM'} at 127.16.43.130:54327
WARNING: no policy specified for kudu/127.16.43.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.43.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.43.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.131.keytab.
Entry for principal kudu/127.16.43.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.131.keytab.
I20260620 09:26:24.354910 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781947562949448-16558-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781947562949448-16558-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781947562949448-16558-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781947562949448-16558-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.43.131:0
--local_ip_for_outbound_sockets=127.16.43.131
--webserver_interface=127.16.43.131
--webserver_port=0
--tserver_master_addrs=127.16.43.190:35973
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.131.keytab
--principal=kudu/127.16.43.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.43.148:37383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781947562949448-16558-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781947562949448-16558-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781947562949448-16558-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:26:24.440039 20975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:26:24.440282 20975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:26:24.440322 20975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:26:24.441572 20975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:26:24.441653 20975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:26:24.441701 20975 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.43.131
I20260620 09:26:24.443277 20975 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:37383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781947562949448-16558-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781947562949448-16558-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.131.keytab
--principal=kudu/127.16.43.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.43.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781947562949448-16558-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.43.131
--webserver_port=0
--tserver_master_addrs=127.16.43.190:35973
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20975
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.43.131
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781947562949448-16558-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:26:24.443580 20975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:26:24.443846 20975 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:26:24.449163 20980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:26:24.449148 20981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:26:24.449194 20983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:26:24.449476 20975 server_base.cc:1034] running on GCE node
I20260620 09:26:24.449708 20975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:26:24.449947 20975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:26:24.451098 20975 hybrid_clock.cc:648] HybridClock initialized: now 1781947584451080 us; error 34 us; skew 500 ppm
Jun 20 09:26:24 dist-test-slave-706q krb5kdc[20603](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947584, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:26:24.454175 20975 init.cc:377] Logged in from keytab as kudu/127.16.43.131@KRBTEST.COM (short username kudu)
I20260620 09:26:24.454481 20975 fs_manager.cc:359] Metadata directory not provided
I20260620 09:26:24.454527 20975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:26:24.454670 20975 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:26:24.455804 20975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781947562949448-16558-0/minicluster-data/ts-2/data/instance:
uuid: "2dd2047438864946bea08045d5285e58"
format_stamp: "Formatted at 2026-06-20 09:26:24 on dist-test-slave-706q"
server_key: "3edb952128803daf5365e530827ce9f3"
server_key_iv: "95ff6a905e71cd15a1b37a1accd7155a"
server_key_version: "encryptionkey@0"
I20260620 09:26:24.456115 20975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781947562949448-16558-0/minicluster-data/ts-2/wal/instance:
uuid: "2dd2047438864946bea08045d5285e58"
format_stamp: "Formatted at 2026-06-20 09:26:24 on dist-test-slave-706q"
server_key: "3edb952128803daf5365e530827ce9f3"
server_key_iv: "95ff6a905e71cd15a1b37a1accd7155a"
server_key_version: "encryptionkey@0"
I20260620 09:26:24.457338 20975 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:26:24.458045 20989 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:24.458276 20975 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:26:24.458364 20975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781947562949448-16558-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781947562949448-16558-0/minicluster-data/ts-2/wal
uuid: "2dd2047438864946bea08045d5285e58"
format_stamp: "Formatted at 2026-06-20 09:26:24 on dist-test-slave-706q"
server_key: "3edb952128803daf5365e530827ce9f3"
server_key_iv: "95ff6a905e71cd15a1b37a1accd7155a"
server_key_version: "encryptionkey@0"
I20260620 09:26:24.458456 20975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781947562949448-16558-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781947562949448-16558-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781947562949448-16558-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:26:24.470774 20975 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:26:24.473147 20975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:26:24.473299 20975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:26:24.473551 20975 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:26:24.473934 20975 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:26:24.473970 20975 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:24.473994 20975 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:26:24.474006 20975 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:24.479784 20975 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.131:42589
I20260620 09:26:24.479866 21102 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.131:42589 every 8 connection(s)
I20260620 09:26:24.480140 20975 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781947562949448-16558-0/minicluster-data/ts-2/data/info.pb
I20260620 09:26:24.480502 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 20975
I20260620 09:26:24.480661 16558 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781947562949448-16558-0/minicluster-data/ts-2/wal/instance
I20260620 09:26:24.480856 16558 external_mini_cluster.cc:1428] Setting key 14f1bf0b02aa1785794fcf1aa856c3d9
Jun 20 09:26:24 dist-test-slave-706q krb5kdc[20603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947584, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.131@KRBTEST.COM for kudu/127.16.43.190@KRBTEST.COM
I20260620 09:26:24.489847 20688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:24.480955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.131:46489 (local address 127.16.43.190:35973)
0620 09:26:24.481027 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:24.481030 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:24.481304 (+ 274us) server_negotiation.cc:407] Connection header received
0620 09:26:24.481962 (+ 658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:24.481963 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:24.481972 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:24.482018 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:24.482923 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:24.483434 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:24.484508 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:24.484707 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:24.484779 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:24.486943 (+ 2164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:24.486945 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:24.486945 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:26:24.486952 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:24.488425 (+ 1473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:24.488824 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:24.488825 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:24.488826 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:24.488889 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:24.489167 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:24.489168 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:24.489168 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:24.489299 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:24.489370 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:24.489638 (+ 268us) server_negotiation.cc:299] Negotiation successful
0620 09:26:24.489713 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 09:26:24.489905 21105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:24.480926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:35973 (local address 127.16.43.131:46489)
0620 09:26:24.481156 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:24.481207 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:24.481867 (+ 660us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:24.482074 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:24.482077 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:24.482341 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:24.482841 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:24.482846 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:24.483513 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:24.483513 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:24.484365 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:24.484367 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:24.484791 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:24.484792 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:24.484896 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:24.485468 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:26:24.485477 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:26:24.486853 (+ 1376us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:26:24.488497 (+ 1644us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:24.488502 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:24.488504 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:24.488738 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:24.488965 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:24.488966 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:24.488966 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:24.489101 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:24.489364 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:26:24.489368 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:26:24.489487 (+ 119us) client_negotiation.cc:769] Sending connection context
0620 09:26:24.489614 (+ 127us) client_negotiation.cc:240] Negotiation successful
0620 09:26:24.489741 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":103,"threads_started":1}
I20260620 09:26:24.490553 21103 heartbeater.cc:347] Connected to a master server at 127.16.43.190:35973
I20260620 09:26:24.490688 21103 heartbeater.cc:464] Registering TS with master...
I20260620 09:26:24.490856 21103 heartbeater.cc:511] Master 127.16.43.190:35973 requested a full tablet report, sending...
I20260620 09:26:24.491395 20646 ts_manager.cc:194] Registered new tserver with Master: 2dd2047438864946bea08045d5285e58 (127.16.43.131:42589)
I20260620 09:26:24.491784 20646 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.43.131@KRBTEST.COM'} at 127.16.43.131:46489
I20260620 09:26:24.492609 16558 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 09:26:24.500082 20688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:24.493956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41652 (local address 127.16.43.190:35973)
0620 09:26:24.494000 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:24.494002 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:24.494142 (+ 140us) server_negotiation.cc:407] Connection header received
0620 09:26:24.494179 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:24.494180 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:24.494188 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:24.494226 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:24.494772 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:24.495237 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:24.495986 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:24.496177 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:24.496234 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:24.497403 (+ 1169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:24.497404 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:24.497404 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:26:24.497410 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:24.498779 (+ 1369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:24.499134 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:24.499134 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:24.499135 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:24.499184 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:24.499381 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:24.499382 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:24.499382 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:24.499520 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:24.499717 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:24.499896 (+ 179us) server_negotiation.cc:299] Negotiation successful
0620 09:26:24.499979 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 09:26:24.501775 20646 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41652:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 09:26:24.503394 20646 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 09:26:24.510452 21111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:24.506076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.129:33733 (local address 127.0.0.1:45344)
0620 09:26:24.506491 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:24.506501 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:24.506758 (+ 257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:24.506964 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:24.506965 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:24.506968 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:26:24.507073 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:24.507076 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:24.508494 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:24.508495 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:24.509367 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:24.509370 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:24.510156 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:24.510157 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:24.510232 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:24.510246 (+ 14us) client_negotiation.cc:769] Sending connection context
0620 09:26:24.510318 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 09:26:24.510367 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":401,"mutex_wait_us":71,"thread_start_us":71,"threads_started":1}
I20260620 09:26:24.510461 21113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:24.506272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45344 (local address 127.16.43.129:33733)
0620 09:26:24.506673 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:24.506677 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:24.506688 (+ 11us) server_negotiation.cc:407] Connection header received
0620 09:26:24.506858 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:24.506860 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:24.506875 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:24.506965 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:26:24.507152 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:24.508369 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:24.509466 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:24.510067 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:24.510156 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:24.510221 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:24.510340 (+ 119us) server_negotiation.cc:299] Negotiation successful
0620 09:26:24.510377 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":101,"threads_started":1}
I20260620 09:26:24.511034 21112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:24.506076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.131:42589 (local address 127.0.0.1:35502)
0620 09:26:24.506257 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:24.506301 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:24.506390 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:24.506947 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:24.506950 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:24.506953 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:26:24.507042 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:24.507045 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:24.508449 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:24.508450 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:24.509772 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:24.509774 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:24.510709 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:24.510710 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:24.510778 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:24.510783 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 09:26:24.510902 (+ 119us) client_negotiation.cc:240] Negotiation successful
0620 09:26:24.510944 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"mutex_wait_us":7,"spinlock_wait_cycles":1280,"thread_start_us":76,"threads_started":1}
I20260620 09:26:24.511145 21114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:24.506335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.130:40553 (local address 127.0.0.1:53658)
0620 09:26:24.506737 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:24.506747 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:24.506800 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:24.507144 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:24.507145 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:24.507147 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:26:24.507180 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:24.507181 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:24.508938 (+ 1757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:24.508939 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:24.509973 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:24.509975 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:24.510922 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:24.510923 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:24.510973 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:24.510976 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 09:26:24.511020 (+ 44us) client_negotiation.cc:240] Negotiation successful
0620 09:26:24.511067 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":51,"threads_started":1}
I20260620 09:26:24.511307 21116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:24.506478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35502 (local address 127.16.43.131:42589)
0620 09:26:24.506739 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:24.506743 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:24.506752 (+ 9us) server_negotiation.cc:407] Connection header received
0620 09:26:24.506791 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:24.506794 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:24.506810 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:24.507122 (+ 312us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:26:24.507200 (+ 78us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:24.508339 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:24.509863 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:24.510595 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:24.510702 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:24.510753 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:24.511189 (+ 436us) server_negotiation.cc:299] Negotiation successful
0620 09:26:24.511238 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":79,"threads_started":1}
I20260620 09:26:24.511356 21115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:24.506462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53658 (local address 127.16.43.130:40553)
0620 09:26:24.506661 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:24.506667 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:24.506867 (+ 200us) server_negotiation.cc:407] Connection header received
0620 09:26:24.506949 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:24.506952 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:24.506969 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:24.507053 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:26:24.507267 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:24.508833 (+ 1566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:24.510066 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:24.510768 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:24.510835 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:24.510891 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:24.511235 (+ 344us) server_negotiation.cc:299] Negotiation successful
0620 09:26:24.511275 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":90,"threads_started":1}
I20260620 09:26:24.511641 20771 tablet_service.cc:1467] Processing CreateTablet for tablet 03e13970842e4d93acc26a0d7f45c4b8 (DEFAULT_TABLE table=test-table [id=fcd4ab503df1475a89d6b7b42bb2df0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:26:24.512004 20771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03e13970842e4d93acc26a0d7f45c4b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:26:24.511931 20904 tablet_service.cc:1467] Processing CreateTablet for tablet 03e13970842e4d93acc26a0d7f45c4b8 (DEFAULT_TABLE table=test-table [id=fcd4ab503df1475a89d6b7b42bb2df0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:26:24.511950 21037 tablet_service.cc:1467] Processing CreateTablet for tablet 03e13970842e4d93acc26a0d7f45c4b8 (DEFAULT_TABLE table=test-table [id=fcd4ab503df1475a89d6b7b42bb2df0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:26:24.512189 20904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03e13970842e4d93acc26a0d7f45c4b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:26:24.512241 21037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03e13970842e4d93acc26a0d7f45c4b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:26:24.514662 21117 tablet_bootstrap.cc:492] T 03e13970842e4d93acc26a0d7f45c4b8 P 6f161ceded5e47f7bf91e999713bc016: Bootstrap starting.
I20260620 09:26:24.515312 21117 tablet_bootstrap.cc:654] T 03e13970842e4d93acc26a0d7f45c4b8 P 6f161ceded5e47f7bf91e999713bc016: Neither blocks nor log segments found. Creating new log.
I20260620 09:26:24.515322 21119 tablet_bootstrap.cc:492] T 03e13970842e4d93acc26a0d7f45c4b8 P dab78762a30e450fab9339720be18fe3: Bootstrap starting.
I20260620 09:26:24.515982 21117 log.cc:824] T 03e13970842e4d93acc26a0d7f45c4b8 P 6f161ceded5e47f7bf91e999713bc016: Log is configured to *not* fsync() on all Append() calls
I20260620 09:26:24.516039 21119 tablet_bootstrap.cc:654] T 03e13970842e4d93acc26a0d7f45c4b8 P dab78762a30e450fab9339720be18fe3: Neither blocks nor log segments found. Creating new log.
I20260620 09:26:24.516630 21119 log.cc:824] T 03e13970842e4d93acc26a0d7f45c4b8 P dab78762a30e450fab9339720be18fe3: Log is configured to *not* fsync() on all Append() calls
I20260620 09:26:24.516832 21117 tablet_bootstrap.cc:492] T 03e13970842e4d93acc26a0d7f45c4b8 P 6f161ceded5e47f7bf91e999713bc016: No bootstrap required, opened a new log
I20260620 09:26:24.516911 21117 ts_tablet_manager.cc:1397] T 03e13970842e4d93acc26a0d7f45c4b8 P 6f161ceded5e47f7bf91e999713bc016: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 09:26:24.516968 21118 tablet_bootstrap.cc:492] T 03e13970842e4d93acc26a0d7f45c4b8 P 2dd2047438864946bea08045d5285e58: Bootstrap starting.
I20260620 09:26:24.517592 21118 tablet_bootstrap.cc:654] T 03e13970842e4d93acc26a0d7f45c4b8 P 2dd2047438864946bea08045d5285e58: Neither blocks nor log segments found. Creating new log.
I20260620 09:26:24.518157 21118 log.cc:824] T 03e13970842e4d93acc26a0d7f45c4b8 P 2dd2047438864946bea08045d5285e58: Log is configured to *not* fsync() on all Append() calls
I20260620 09:26:24.518677 21119 tablet_bootstrap.cc:492] T 03e13970842e4d93acc26a0d7f45c4b8 P dab78762a30e450fab9339720be18fe3: No bootstrap required, opened a new log
I20260620 09:26:24.518745 21119 ts_tablet_manager.cc:1397] T 03e13970842e4d93acc26a0d7f45c4b8 P dab78762a30e450fab9339720be18fe3: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 09:26:24.518975 21118 tablet_bootstrap.cc:492] T 03e13970842e4d93acc26a0d7f45c4b8 P 2dd2047438864946bea08045d5285e58: No bootstrap required, opened a new log
I20260620 09:26:24.519063 21118 ts_tablet_manager.cc:1397] T 03e13970842e4d93acc26a0d7f45c4b8 P 2dd2047438864946bea08045d5285e58: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 09:26:24.519192 21117 raft_consensus.cc:348] T 03e13970842e4d93acc26a0d7f45c4b8 P 6f161ceded5e47f7bf91e999713bc016 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dd2047438864946bea08045d5285e58" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 42589 } } peers { permanent_uuid: "dab78762a30e450fab9339720be18fe3" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 33733 } } peers { permanent_uuid: "6f161ceded5e47f7bf91e999713bc016" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 40553 } }
I20260620 09:26:24.519354 21117 raft_consensus.cc:374] T 03e13970842e4d93acc26a0d7f45c4b8 P 6f161ceded5e47f7bf91e999713bc016 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:26:24.519385 21117 raft_consensus.cc:729] T 03e13970842e4d93acc26a0d7f45c4b8 P 6f161ceded5e47f7bf91e999713bc016 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f161ceded5e47f7bf91e999713bc016, State: Initialized, Role: FOLLOWER
I20260620 09:26:24.519505 21117 consensus_queue.cc:260] T 03e13970842e4d93acc26a0d7f45c4b8 P 6f161ceded5e47f7bf91e999713bc016 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dd2047438864946bea08045d5285e58" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 42589 } } peers { permanent_uuid: "dab78762a30e450fab9339720be18fe3" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 33733 } } peers { permanent_uuid: "6f161ceded5e47f7bf91e999713bc016" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 40553 } }
I20260620 09:26:24.519824 21117 ts_tablet_manager.cc:1428] T 03e13970842e4d93acc26a0d7f45c4b8 P 6f161ceded5e47f7bf91e999713bc016: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260620 09:26:24.519837 20970 heartbeater.cc:503] Master 127.16.43.190:35973 was elected leader, sending a full tablet report...
I20260620 09:26:24.520687 21119 raft_consensus.cc:348] T 03e13970842e4d93acc26a0d7f45c4b8 P dab78762a30e450fab9339720be18fe3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dd2047438864946bea08045d5285e58" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 42589 } } peers { permanent_uuid: "dab78762a30e450fab9339720be18fe3" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 33733 } } peers { permanent_uuid: "6f161ceded5e47f7bf91e999713bc016" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 40553 } }
I20260620 09:26:24.520862 21119 raft_consensus.cc:374] T 03e13970842e4d93acc26a0d7f45c4b8 P dab78762a30e450fab9339720be18fe3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:26:24.520910 21119 raft_consensus.cc:729] T 03e13970842e4d93acc26a0d7f45c4b8 P dab78762a30e450fab9339720be18fe3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dab78762a30e450fab9339720be18fe3, State: Initialized, Role: FOLLOWER
I20260620 09:26:24.521047 21119 consensus_queue.cc:260] T 03e13970842e4d93acc26a0d7f45c4b8 P dab78762a30e450fab9339720be18fe3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dd2047438864946bea08045d5285e58" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 42589 } } peers { permanent_uuid: "dab78762a30e450fab9339720be18fe3" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 33733 } } peers { permanent_uuid: "6f161ceded5e47f7bf91e999713bc016" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 40553 } }
I20260620 09:26:24.521364 21119 ts_tablet_manager.cc:1428] T 03e13970842e4d93acc26a0d7f45c4b8 P dab78762a30e450fab9339720be18fe3: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 09:26:24.521382 20837 heartbeater.cc:503] Master 127.16.43.190:35973 was elected leader, sending a full tablet report...
I20260620 09:26:24.521281 21118 raft_consensus.cc:348] T 03e13970842e4d93acc26a0d7f45c4b8 P 2dd2047438864946bea08045d5285e58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dd2047438864946bea08045d5285e58" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 42589 } } peers { permanent_uuid: "dab78762a30e450fab9339720be18fe3" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 33733 } } peers { permanent_uuid: "6f161ceded5e47f7bf91e999713bc016" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 40553 } }
I20260620 09:26:24.521438 21118 raft_consensus.cc:374] T 03e13970842e4d93acc26a0d7f45c4b8 P 2dd2047438864946bea08045d5285e58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:26:24.521474 21118 raft_consensus.cc:729] T 03e13970842e4d93acc26a0d7f45c4b8 P 2dd2047438864946bea08045d5285e58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2dd2047438864946bea08045d5285e58, State: Initialized, Role: FOLLOWER
I20260620 09:26:24.521590 21118 consensus_queue.cc:260] T 03e13970842e4d93acc26a0d7f45c4b8 P 2dd2047438864946bea08045d5285e58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dd2047438864946bea08045d5285e58" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 42589 } } peers { permanent_uuid: "dab78762a30e450fab9339720be18fe3" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 33733 } } peers { permanent_uuid: "6f161ceded5e47f7bf91e999713bc016" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 40553 } }
I20260620 09:26:24.521889 21118 ts_tablet_manager.cc:1428] T 03e13970842e4d93acc26a0d7f45c4b8 P 2dd2047438864946bea08045d5285e58: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 09:26:24.521900 21103 heartbeater.cc:503] Master 127.16.43.190:35973 was elected leader, sending a full tablet report...
W20260620 09:26:24.574312 20971 tablet.cc:2369] T 03e13970842e4d93acc26a0d7f45c4b8 P 6f161ceded5e47f7bf91e999713bc016: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 09:26:24.633297 21125 raft_consensus.cc:482] T 03e13970842e4d93acc26a0d7f45c4b8 P 2dd2047438864946bea08045d5285e58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:26:24.633430 21125 raft_consensus.cc:504] T 03e13970842e4d93acc26a0d7f45c4b8 P 2dd2047438864946bea08045d5285e58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dd2047438864946bea08045d5285e58" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 42589 } } peers { permanent_uuid: "dab78762a30e450fab9339720be18fe3" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 33733 } } peers { permanent_uuid: "6f161ceded5e47f7bf91e999713bc016" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 40553 } }
I20260620 09:26:24.633767 21125 leader_election.cc:288] T 03e13970842e4d93acc26a0d7f45c4b8 P 2dd2047438864946bea08045d5285e58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dab78762a30e450fab9339720be18fe3 (127.16.43.129:33733), 6f161ceded5e47f7bf91e999713bc016 (127.16.43.130:40553)
I20260620 09:26:24.637101 21105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:24.633943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.129:33733 (local address 127.16.43.131:50109)
0620 09:26:24.634019 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:24.634032 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:24.634078 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:24.634322 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:24.634323 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:24.634326 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:26:24.634417 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:24.634420 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:24.635248 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:24.635249 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:24.636219 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:24.636221 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:24.636820 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:24.636821 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:24.636906 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:24.636910 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 09:26:24.636963 (+ 53us) client_negotiation.cc:240] Negotiation successful
0620 09:26:24.637008 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260620 09:26:24.637132 21113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:24.634038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.131:50109 (local address 127.16.43.129:33733)
0620 09:26:24.634156 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:24.634158 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:24.634170 (+ 12us) server_negotiation.cc:407] Connection header received
0620 09:26:24.634204 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:24.634205 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:24.634217 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:24.634298 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:26:24.634611 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:24.635135 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:24.636336 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:24.636729 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:24.636787 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:24.636794 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:24.636985 (+ 191us) server_negotiation.cc:299] Negotiation successful
0620 09:26:24.637035 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260620 09:26:24.637223 21115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:24.634193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.131:54063 (local address 127.16.43.130:40553)
0620 09:26:24.634242 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:24.634244 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:24.634374 (+ 130us) server_negotiation.cc:407] Connection header received
0620 09:26:24.634410 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:24.634411 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:24.634420 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:24.634481 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:26:24.634743 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:24.635294 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:24.636407 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:24.636829 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:24.636903 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:24.636911 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:24.637091 (+ 180us) server_negotiation.cc:299] Negotiation successful
0620 09:26:24.637136 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 09:26:24.637266 21126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:24.634077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.130:40553 (local address 127.16.43.131:54063)
0620 09:26:24.634293 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:24.634305 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:24.634364 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:24.634497 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:24.634498 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:24.634499 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:26:24.634632 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:24.634635 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:24.635471 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:24.635472 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:24.636313 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:24.636317 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:24.636916 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:24.636916 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:24.637007 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:24.637011 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 09:26:24.637068 (+ 57us) client_negotiation.cc:240] Negotiation successful
0620 09:26:24.637103 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"mutex_wait_us":13,"thread_start_us":124,"threads_started":1}
I20260620 09:26:24.637493 20791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03e13970842e4d93acc26a0d7f45c4b8" candidate_uuid: "2dd2047438864946bea08045d5285e58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dab78762a30e450fab9339720be18fe3" is_pre_election: true
I20260620 09:26:24.637594 20924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03e13970842e4d93acc26a0d7f45c4b8" candidate_uuid: "2dd2047438864946bea08045d5285e58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f161ceded5e47f7bf91e999713bc016" is_pre_election: true
I20260620 09:26:24.637676 20791 raft_consensus.cc:2454] T 03e13970842e4d93acc26a0d7f45c4b8 P dab78762a30e450fab9339720be18fe3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2dd2047438864946bea08045d5285e58 in term 0.
I20260620 09:26:24.637709 20924 raft_consensus.cc:2454] T 03e13970842e4d93acc26a0d7f45c4b8 P 6f161ceded5e47f7bf91e999713bc016 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2dd2047438864946bea08045d5285e58 in term 0.
I20260620 09:26:24.637910 20993 leader_election.cc:302] T 03e13970842e4d93acc26a0d7f45c4b8 P 2dd2047438864946bea08045d5285e58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2dd2047438864946bea08045d5285e58, dab78762a30e450fab9339720be18fe3; no voters:
I20260620 09:26:24.638039 21125 raft_consensus.cc:2781] T 03e13970842e4d93acc26a0d7f45c4b8 P 2dd2047438864946bea08045d5285e58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 09:26:24.638093 21125 raft_consensus.cc:482] T 03e13970842e4d93acc26a0d7f45c4b8 P 2dd2047438864946bea08045d5285e58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 09:26:24.638123 21125 raft_consensus.cc:3037] T 03e13970842e4d93acc26a0d7f45c4b8 P 2dd2047438864946bea08045d5285e58 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:26:24.638801 21125 raft_consensus.cc:504] T 03e13970842e4d93acc26a0d7f45c4b8 P 2dd2047438864946bea08045d5285e58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dd2047438864946bea08045d5285e58" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 42589 } } peers { permanent_uuid: "dab78762a30e450fab9339720be18fe3" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 33733 } } peers { permanent_uuid: "6f161ceded5e47f7bf91e999713bc016" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 40553 } }
I20260620 09:26:24.638934 21125 leader_election.cc:288] T 03e13970842e4d93acc26a0d7f45c4b8 P 2dd2047438864946bea08045d5285e58 [CANDIDATE]: Term 1 election: Requested vote from peers dab78762a30e450fab9339720be18fe3 (127.16.43.129:33733), 6f161ceded5e47f7bf91e999713bc016 (127.16.43.130:40553)
I20260620 09:26:24.639089 20791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03e13970842e4d93acc26a0d7f45c4b8" candidate_uuid: "2dd2047438864946bea08045d5285e58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dab78762a30e450fab9339720be18fe3"
I20260620 09:26:24.639147 20924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03e13970842e4d93acc26a0d7f45c4b8" candidate_uuid: "2dd2047438864946bea08045d5285e58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f161ceded5e47f7bf91e999713bc016"
I20260620 09:26:24.639209 20791 raft_consensus.cc:3037] T 03e13970842e4d93acc26a0d7f45c4b8 P dab78762a30e450fab9339720be18fe3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:26:24.639216 20924 raft_consensus.cc:3037] T 03e13970842e4d93acc26a0d7f45c4b8 P 6f161ceded5e47f7bf91e999713bc016 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:26:24.639999 20924 raft_consensus.cc:2454] T 03e13970842e4d93acc26a0d7f45c4b8 P 6f161ceded5e47f7bf91e999713bc016 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2dd2047438864946bea08045d5285e58 in term 1.
I20260620 09:26:24.640091 20791 raft_consensus.cc:2454] T 03e13970842e4d93acc26a0d7f45c4b8 P dab78762a30e450fab9339720be18fe3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2dd2047438864946bea08045d5285e58 in term 1.
I20260620 09:26:24.640213 20993 leader_election.cc:302] T 03e13970842e4d93acc26a0d7f45c4b8 P 2dd2047438864946bea08045d5285e58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2dd2047438864946bea08045d5285e58, 6f161ceded5e47f7bf91e999713bc016; no voters:
I20260620 09:26:24.640347 21125 raft_consensus.cc:2781] T 03e13970842e4d93acc26a0d7f45c4b8 P 2dd2047438864946bea08045d5285e58 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:26:24.640504 21125 raft_consensus.cc:686] T 03e13970842e4d93acc26a0d7f45c4b8 P 2dd2047438864946bea08045d5285e58 [term 1 LEADER]: Becoming Leader. State: Replica: 2dd2047438864946bea08045d5285e58, State: Running, Role: LEADER
I20260620 09:26:24.640607 21125 consensus_queue.cc:237] T 03e13970842e4d93acc26a0d7f45c4b8 P 2dd2047438864946bea08045d5285e58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dd2047438864946bea08045d5285e58" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 42589 } } peers { permanent_uuid: "dab78762a30e450fab9339720be18fe3" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 33733 } } peers { permanent_uuid: "6f161ceded5e47f7bf91e999713bc016" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 40553 } }
I20260620 09:26:24.641513 20647 catalog_manager.cc:5495] T 03e13970842e4d93acc26a0d7f45c4b8 P 2dd2047438864946bea08045d5285e58 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2dd2047438864946bea08045d5285e58 (127.16.43.131). New cstate: current_term: 1 leader_uuid: "2dd2047438864946bea08045d5285e58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dd2047438864946bea08045d5285e58" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 42589 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dab78762a30e450fab9339720be18fe3" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 33733 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6f161ceded5e47f7bf91e999713bc016" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 40553 } health_report { overall_health: UNKNOWN } } }
I20260620 09:26:24.657706 21116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:24.654986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35508 (local address 127.16.43.131:42589)
0620 09:26:24.655058 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:24.655060 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:24.655069 (+ 9us) server_negotiation.cc:407] Connection header received
0620 09:26:24.655104 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:24.655104 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:24.655124 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:24.655185 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:26:24.655315 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:24.655913 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:24.656697 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:24.656910 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:24.656997 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:24.657153 (+ 156us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:26:24.657266 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:26:24.657335 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:24.657458 (+ 123us) server_negotiation.cc:299] Negotiation successful
0620 09:26:24.657574 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 09:26:24.659710 20924 raft_consensus.cc:1261] T 03e13970842e4d93acc26a0d7f45c4b8 P 6f161ceded5e47f7bf91e999713bc016 [term 1 FOLLOWER]: Refusing update from remote peer 2dd2047438864946bea08045d5285e58: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:26:24.659709 20791 raft_consensus.cc:1261] T 03e13970842e4d93acc26a0d7f45c4b8 P dab78762a30e450fab9339720be18fe3 [term 1 FOLLOWER]: Refusing update from remote peer 2dd2047438864946bea08045d5285e58: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:26:24.660104 21127 consensus_queue.cc:1035] T 03e13970842e4d93acc26a0d7f45c4b8 P 2dd2047438864946bea08045d5285e58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f161ceded5e47f7bf91e999713bc016" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:26:24.660239 21125 consensus_queue.cc:1035] T 03e13970842e4d93acc26a0d7f45c4b8 P 2dd2047438864946bea08045d5285e58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dab78762a30e450fab9339720be18fe3" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 33733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:26:24.662887 21130 mvcc.cc:205] Tried to move back new op lower bound from 7298857306763014144 to 7298857306688618496. Current Snapshot: MvccSnapshot[applied={T|T < 7298857306763014144}]
I20260620 09:26:24.663059 21132 mvcc.cc:205] Tried to move back new op lower bound from 7298857306763014144 to 7298857306688618496. Current Snapshot: MvccSnapshot[applied={T|T < 7298857306763014144}]
I20260620 09:26:24.663367 21131 mvcc.cc:205] Tried to move back new op lower bound from 7298857306763014144 to 7298857306688618496. Current Snapshot: MvccSnapshot[applied={T|T < 7298857306763014144}]
I20260620 09:26:24.665884 20646 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41652:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:26:24.666015 20646 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41652:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:26:24.666931 20646 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d3d9bbbc9a084b8a9b9ba26971b72e0e: Sending DeleteTablet for 3 replicas of tablet 03e13970842e4d93acc26a0d7f45c4b8
I20260620 09:26:24.667340 20904 tablet_service.cc:1514] Processing DeleteTablet for tablet 03e13970842e4d93acc26a0d7f45c4b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:26:24 UTC) from {username='kudu', principal='kudu/127.16.43.190@KRBTEST.COM'} at 127.0.0.1:53658
I20260620 09:26:24.667344 21037 tablet_service.cc:1514] Processing DeleteTablet for tablet 03e13970842e4d93acc26a0d7f45c4b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:26:24 UTC) from {username='kudu', principal='kudu/127.16.43.190@KRBTEST.COM'} at 127.0.0.1:35502
I20260620 09:26:24.667373 20771 tablet_service.cc:1514] Processing DeleteTablet for tablet 03e13970842e4d93acc26a0d7f45c4b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:26:24 UTC) from {username='kudu', principal='kudu/127.16.43.190@KRBTEST.COM'} at 127.0.0.1:45344
I20260620 09:26:24.667693 21141 tablet_replica.cc:330] T 03e13970842e4d93acc26a0d7f45c4b8 P 2dd2047438864946bea08045d5285e58: stopping tablet replica
I20260620 09:26:24.667785 21142 tablet_replica.cc:330] T 03e13970842e4d93acc26a0d7f45c4b8 P dab78762a30e450fab9339720be18fe3: stopping tablet replica
I20260620 09:26:24.667809 21141 raft_consensus.cc:2229] T 03e13970842e4d93acc26a0d7f45c4b8 P 2dd2047438864946bea08045d5285e58 [term 1 LEADER]: Raft consensus shutting down.
I20260620 09:26:24.667865 21142 raft_consensus.cc:2229] T 03e13970842e4d93acc26a0d7f45c4b8 P dab78762a30e450fab9339720be18fe3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:26:24.667958 21142 raft_consensus.cc:2258] T 03e13970842e4d93acc26a0d7f45c4b8 P dab78762a30e450fab9339720be18fe3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:26:24.668005 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 20709
I20260620 09:26:24.667944 21141 raft_consensus.cc:2258] T 03e13970842e4d93acc26a0d7f45c4b8 P 2dd2047438864946bea08045d5285e58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:26:24.668994 21140 tablet_replica.cc:330] T 03e13970842e4d93acc26a0d7f45c4b8 P 6f161ceded5e47f7bf91e999713bc016: stopping tablet replica
I20260620 09:26:24.669098 21140 raft_consensus.cc:2229] T 03e13970842e4d93acc26a0d7f45c4b8 P 6f161ceded5e47f7bf91e999713bc016 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:26:24.669183 21140 raft_consensus.cc:2258] T 03e13970842e4d93acc26a0d7f45c4b8 P 6f161ceded5e47f7bf91e999713bc016 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:26:24.669520 21141 ts_tablet_manager.cc:1905] T 03e13970842e4d93acc26a0d7f45c4b8 P 2dd2047438864946bea08045d5285e58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:26:24.670969 21141 ts_tablet_manager.cc:1918] T 03e13970842e4d93acc26a0d7f45c4b8 P 2dd2047438864946bea08045d5285e58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:26:24.671056 21141 log.cc:1196] T 03e13970842e4d93acc26a0d7f45c4b8 P 2dd2047438864946bea08045d5285e58: Deleting WAL directory at /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781947562949448-16558-0/minicluster-data/ts-2/wal/wals/03e13970842e4d93acc26a0d7f45c4b8
I20260620 09:26:24.671376 21141 ts_tablet_manager.cc:1939] T 03e13970842e4d93acc26a0d7f45c4b8 P 2dd2047438864946bea08045d5285e58: Deleting consensus metadata
I20260620 09:26:24.671869 20631 catalog_manager.cc:4841] TS 2dd2047438864946bea08045d5285e58 (127.16.43.131:42589): tablet 03e13970842e4d93acc26a0d7f45c4b8 (table test-table [id=fcd4ab503df1475a89d6b7b42bb2df0c]) successfully deleted
I20260620 09:26:24.672257 21140 ts_tablet_manager.cc:1905] T 03e13970842e4d93acc26a0d7f45c4b8 P 6f161ceded5e47f7bf91e999713bc016: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:26:24.673676 21140 ts_tablet_manager.cc:1918] T 03e13970842e4d93acc26a0d7f45c4b8 P 6f161ceded5e47f7bf91e999713bc016: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:26:24.673807 21140 log.cc:1196] T 03e13970842e4d93acc26a0d7f45c4b8 P 6f161ceded5e47f7bf91e999713bc016: Deleting WAL directory at /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781947562949448-16558-0/minicluster-data/ts-1/wal/wals/03e13970842e4d93acc26a0d7f45c4b8
W20260620 09:26:24.674083 20634 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.43.129:33733 (error 108)
I20260620 09:26:24.674235 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 20842
I20260620 09:26:24.676047 21114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:24.675893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.129:33733 (local address 127.0.0.1:45360)
0620 09:26:24.675957 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:24.676008 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.129:33733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260620 09:26:24.676271 20634 catalog_manager.cc:4584] TS dab78762a30e450fab9339720be18fe3 (127.16.43.129:33733): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 03e13970842e4d93acc26a0d7f45c4b8: Network error: Client connection negotiation failed: client connection to 127.16.43.129:33733: connect: Connection refused (error 111)
I20260620 09:26:24.683310 21114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:24.683174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.130:40553 (local address 127.0.0.1:53674)
0620 09:26:24.683243 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:24.683273 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.130:40553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 09:26:24.683296 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 20975
I20260620 09:26:24.688675 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 20616
2026-06-20T09:26:24Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2008 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-20T09:26:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:26:24Z Disabled control of system clock
I20260620 09:26:24.716118 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781947562949448-16558-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781947562949448-16558-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781947562949448-16558-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.43.190:39727
--webserver_interface=127.16.43.190
--webserver_port=0
--builtin_ntp_servers=127.16.43.148:44371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.43.190:39727
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 09:26:24.796859 21147 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:26:24.797075 21147 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:26:24.797096 21147 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:26:24.798321 21147 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:26:24.798373 21147 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:26:24.798393 21147 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:26:24.798416 21147 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:26:24.798429 21147 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:26:24.799829 21147 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:44371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781947562949448-16558-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781947562949448-16558-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.43.190:39727
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.43.190:39727
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.43.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21147
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781947562949448-16558-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:26:24.800024 21147 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:26:24.800259 21147 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:26:24.805105 21152 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:26:24.805248 21153 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:26:24.805352 21147 server_base.cc:1034] running on GCE node
W20260620 09:26:24.805157 21155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:26:24.805728 21147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:26:24.806059 21147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:26:24.807228 21147 hybrid_clock.cc:648] HybridClock initialized: now 1781947584807217 us; error 46 us; skew 500 ppm
I20260620 09:26:24.808492 21147 webserver.cc:460] Webserver started at http://127.16.43.190:42671/ using document root <none> and password file <none>
I20260620 09:26:24.808755 21147 fs_manager.cc:359] Metadata directory not provided
I20260620 09:26:24.808830 21147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:26:24.808981 21147 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:26:24.810055 21147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781947562949448-16558-0/minicluster-data/master-0/data/instance:
uuid: "cf8d77d815e34f50a03e113fa415cef4"
format_stamp: "Formatted at 2026-06-20 09:26:24 on dist-test-slave-706q"
server_key: "6af9d65058d9745cf7ea8df70ed168cc"
server_key_iv: "f6d2d60c68849963bbf23e5bb56363e6"
server_key_version: "encryptionkey@0"
I20260620 09:26:24.810422 21147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781947562949448-16558-0/minicluster-data/master-0/wal/instance:
uuid: "cf8d77d815e34f50a03e113fa415cef4"
format_stamp: "Formatted at 2026-06-20 09:26:24 on dist-test-slave-706q"
server_key: "6af9d65058d9745cf7ea8df70ed168cc"
server_key_iv: "f6d2d60c68849963bbf23e5bb56363e6"
server_key_version: "encryptionkey@0"
I20260620 09:26:24.811784 21147 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 09:26:24.812476 21161 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:24.812696 21147 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 09:26:24.812803 21147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781947562949448-16558-0/minicluster-data/master-0/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781947562949448-16558-0/minicluster-data/master-0/wal
uuid: "cf8d77d815e34f50a03e113fa415cef4"
format_stamp: "Formatted at 2026-06-20 09:26:24 on dist-test-slave-706q"
server_key: "6af9d65058d9745cf7ea8df70ed168cc"
server_key_iv: "f6d2d60c68849963bbf23e5bb56363e6"
server_key_version: "encryptionkey@0"
I20260620 09:26:24.812878 21147 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781947562949448-16558-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781947562949448-16558-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781947562949448-16558-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:26:24.816403 21147 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:26:24.816695 21147 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:26:24.816838 21147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:26:24.820736 21147 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.190:39727
I20260620 09:26:24.820765 21213 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.190:39727 every 8 connection(s)
I20260620 09:26:24.821092 21147 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
I20260620 09:26:24.821729 21214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:26:24.824503 21214 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf8d77d815e34f50a03e113fa415cef4: Bootstrap starting.
I20260620 09:26:24.825049 21214 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cf8d77d815e34f50a03e113fa415cef4: Neither blocks nor log segments found. Creating new log.
I20260620 09:26:24.825302 21214 log.cc:824] T 00000000000000000000000000000000 P cf8d77d815e34f50a03e113fa415cef4: Log is configured to *not* fsync() on all Append() calls
I20260620 09:26:24.826081 21214 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf8d77d815e34f50a03e113fa415cef4: No bootstrap required, opened a new log
I20260620 09:26:24.827733 21214 raft_consensus.cc:348] T 00000000000000000000000000000000 P cf8d77d815e34f50a03e113fa415cef4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf8d77d815e34f50a03e113fa415cef4" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 39727 } }
I20260620 09:26:24.827867 21214 raft_consensus.cc:374] T 00000000000000000000000000000000 P cf8d77d815e34f50a03e113fa415cef4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:26:24.827889 21214 raft_consensus.cc:729] T 00000000000000000000000000000000 P cf8d77d815e34f50a03e113fa415cef4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf8d77d815e34f50a03e113fa415cef4, State: Initialized, Role: FOLLOWER
I20260620 09:26:24.827989 21214 consensus_queue.cc:260] T 00000000000000000000000000000000 P cf8d77d815e34f50a03e113fa415cef4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf8d77d815e34f50a03e113fa415cef4" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 39727 } }
I20260620 09:26:24.828068 21214 raft_consensus.cc:388] T 00000000000000000000000000000000 P cf8d77d815e34f50a03e113fa415cef4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:26:24.828104 21214 raft_consensus.cc:482] T 00000000000000000000000000000000 P cf8d77d815e34f50a03e113fa415cef4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:26:24.828152 21214 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cf8d77d815e34f50a03e113fa415cef4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:26:24.828672 21214 raft_consensus.cc:504] T 00000000000000000000000000000000 P cf8d77d815e34f50a03e113fa415cef4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf8d77d815e34f50a03e113fa415cef4" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 39727 } }
I20260620 09:26:24.828790 21214 leader_election.cc:302] T 00000000000000000000000000000000 P cf8d77d815e34f50a03e113fa415cef4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cf8d77d815e34f50a03e113fa415cef4; no voters:
I20260620 09:26:24.828962 21214 leader_election.cc:288] T 00000000000000000000000000000000 P cf8d77d815e34f50a03e113fa415cef4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:26:24.829017 21217 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cf8d77d815e34f50a03e113fa415cef4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:26:24.829216 21214 sys_catalog.cc:564] T 00000000000000000000000000000000 P cf8d77d815e34f50a03e113fa415cef4 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:26:24.829216 21217 raft_consensus.cc:686] T 00000000000000000000000000000000 P cf8d77d815e34f50a03e113fa415cef4 [term 1 LEADER]: Becoming Leader. State: Replica: cf8d77d815e34f50a03e113fa415cef4, State: Running, Role: LEADER
I20260620 09:26:24.829344 21217 consensus_queue.cc:237] T 00000000000000000000000000000000 P cf8d77d815e34f50a03e113fa415cef4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf8d77d815e34f50a03e113fa415cef4" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 39727 } }
I20260620 09:26:24.829772 21218 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf8d77d815e34f50a03e113fa415cef4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cf8d77d815e34f50a03e113fa415cef4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf8d77d815e34f50a03e113fa415cef4" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 39727 } } }
I20260620 09:26:24.829856 21218 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf8d77d815e34f50a03e113fa415cef4 [sys.catalog]: This master's current role is: LEADER
I20260620 09:26:24.830106 21219 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf8d77d815e34f50a03e113fa415cef4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cf8d77d815e34f50a03e113fa415cef4. Latest consensus state: current_term: 1 leader_uuid: "cf8d77d815e34f50a03e113fa415cef4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf8d77d815e34f50a03e113fa415cef4" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 39727 } } }
I20260620 09:26:24.830241 21226 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:26:24.830255 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 21147
I20260620 09:26:24.830261 21219 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf8d77d815e34f50a03e113fa415cef4 [sys.catalog]: This master's current role is: LEADER
I20260620 09:26:24.830353 16558 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781947562949448-16558-0/minicluster-data/master-0/wal/instance
I20260620 09:26:24.830529 16558 external_mini_cluster.cc:1428] Setting key 40d3fc7a72f35e76ddc0a7dd24fb42e6
I20260620 09:26:24.830854 21226 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:26:24.832849 21226 catalog_manager.cc:1347] Generated new cluster ID: 697c9c3cc17e46b687a08ae89658cdbb
I20260620 09:26:24.832912 21226 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:26:24.833446 21236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:24.831154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38280 (local address 127.16.43.190:39727)
0620 09:26:24.831766 (+ 612us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:24.831775 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:24.831848 (+ 73us) server_negotiation.cc:407] Connection header received
0620 09:26:24.831973 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:24.831978 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:24.832035 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:24.832184 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:24.832940 (+ 756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:24.832943 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:24.832943 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:26:24.832948 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:24.832996 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:26:24.833011 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:24.833088 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:24.833166 (+ 78us) server_negotiation.cc:299] Negotiation successful
0620 09:26:24.833222 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":539,"thread_start_us":70,"threads_started":1}
I20260620 09:26:24.852228 21226 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:26:24.852460 21226 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:26:24.859551 21226 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cf8d77d815e34f50a03e113fa415cef4: Generated new TSK 0
I20260620 09:26:24.859825 21226 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 09:26:24.862138 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781947562949448-16558-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781947562949448-16558-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781947562949448-16558-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781947562949448-16558-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.43.129:0
--local_ip_for_outbound_sockets=127.16.43.129
--webserver_interface=127.16.43.129
--webserver_port=0
--tserver_master_addrs=127.16.43.190:39727
--builtin_ntp_servers=127.16.43.148:44371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 09:26:24.947883 21238 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:26:24.948140 21238 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:26:24.948185 21238 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:26:24.949417 21238 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:26:24.949496 21238 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:26:24.949539 21238 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.43.129
I20260620 09:26:24.950944 21238 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:44371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781947562949448-16558-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781947562949448-16558-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.43.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781947562949448-16558-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.43.129
--webserver_port=0
--tserver_master_addrs=127.16.43.190:39727
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21238
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.43.129
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781947562949448-16558-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:26:24.951239 21238 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:26:24.951581 21238 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:26:24.957698 21246 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:26:24.957857 21243 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:26:24.957736 21244 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:26:24.958129 21238 server_base.cc:1034] running on GCE node
I20260620 09:26:24.958364 21238 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:26:24.958652 21238 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:26:24.959813 21238 hybrid_clock.cc:648] HybridClock initialized: now 1781947584959795 us; error 33 us; skew 500 ppm
I20260620 09:26:24.961164 21238 webserver.cc:460] Webserver started at http://127.16.43.129:45579/ using document root <none> and password file <none>
I20260620 09:26:24.961405 21238 fs_manager.cc:359] Metadata directory not provided
I20260620 09:26:24.961476 21238 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:26:24.961643 21238 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:26:24.962754 21238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781947562949448-16558-0/minicluster-data/ts-0/data/instance:
uuid: "3c77cefff54242a0a529341503cc96ea"
format_stamp: "Formatted at 2026-06-20 09:26:24 on dist-test-slave-706q"
server_key: "b66f9f3adcd151f375c93a3a12f335e0"
server_key_iv: "e8dceafa38b56eed5dcbfe7bd56655c8"
server_key_version: "encryptionkey@0"
I20260620 09:26:24.963131 21238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781947562949448-16558-0/minicluster-data/ts-0/wal/instance:
uuid: "3c77cefff54242a0a529341503cc96ea"
format_stamp: "Formatted at 2026-06-20 09:26:24 on dist-test-slave-706q"
server_key: "b66f9f3adcd151f375c93a3a12f335e0"
server_key_iv: "e8dceafa38b56eed5dcbfe7bd56655c8"
server_key_version: "encryptionkey@0"
I20260620 09:26:24.964581 21238 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 09:26:24.965378 21252 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:24.965632 21238 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 09:26:24.965730 21238 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781947562949448-16558-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781947562949448-16558-0/minicluster-data/ts-0/wal
uuid: "3c77cefff54242a0a529341503cc96ea"
format_stamp: "Formatted at 2026-06-20 09:26:24 on dist-test-slave-706q"
server_key: "b66f9f3adcd151f375c93a3a12f335e0"
server_key_iv: "e8dceafa38b56eed5dcbfe7bd56655c8"
server_key_version: "encryptionkey@0"
I20260620 09:26:24.965818 21238 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781947562949448-16558-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781947562949448-16558-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781947562949448-16558-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:26:24.970160 21238 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:26:24.970396 21238 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:26:24.970494 21238 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:26:24.970691 21238 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:26:24.970939 21238 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:26:24.970968 21238 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:24.970989 21238 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:26:24.971002 21238 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:24.976649 21238 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.129:39971
I20260620 09:26:24.976750 21365 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.129:39971 every 8 connection(s)
I20260620 09:26:24.977051 21238 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781947562949448-16558-0/minicluster-data/ts-0/data/info.pb
I20260620 09:26:24.977595 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 21238
I20260620 09:26:24.977697 16558 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781947562949448-16558-0/minicluster-data/ts-0/wal/instance
I20260620 09:26:24.977888 16558 external_mini_cluster.cc:1428] Setting key 9c45b510f6fb7bd95fe3101038d91fca
I20260620 09:26:24.978865 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781947562949448-16558-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781947562949448-16558-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781947562949448-16558-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781947562949448-16558-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.43.130:0
--local_ip_for_outbound_sockets=127.16.43.130
--webserver_interface=127.16.43.130
--webserver_port=0
--tserver_master_addrs=127.16.43.190:39727
--builtin_ntp_servers=127.16.43.148:44371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260620 09:26:24.980861 21236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:24.977892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.129:44585 (local address 127.16.43.190:39727)
0620 09:26:24.977959 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:24.977962 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:24.978382 (+ 420us) server_negotiation.cc:407] Connection header received
0620 09:26:24.979143 (+ 761us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:24.979144 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:24.979150 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:24.979201 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:24.980606 (+ 1405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:24.980608 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:24.980608 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:26:24.980613 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:24.980631 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:26:24.980636 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:24.980709 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:24.980754 (+ 45us) server_negotiation.cc:299] Negotiation successful
0620 09:26:24.980772 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 09:26:24.981012 21368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:24.977909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:39727 (local address 127.16.43.129:44585)
0620 09:26:24.978248 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:24.978297 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:24.978978 (+ 681us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:24.979286 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:24.979290 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:24.979301 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:24.980455 (+ 1154us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 09:26:24.980456 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:26:24.980485 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 09:26:24.980514 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:26:24.980704 (+ 190us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:26:24.980706 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:26:24.980708 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 09:26:24.980761 (+ 53us) client_negotiation.cc:240] Negotiation successful
0620 09:26:24.980808 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":106,"threads_started":1}
I20260620 09:26:24.981541 21366 heartbeater.cc:347] Connected to a master server at 127.16.43.190:39727
I20260620 09:26:24.981657 21366 heartbeater.cc:464] Registering TS with master...
I20260620 09:26:24.981776 21366 heartbeater.cc:511] Master 127.16.43.190:39727 requested a full tablet report, sending...
I20260620 09:26:24.982327 21178 ts_manager.cc:194] Registered new tserver with Master: 3c77cefff54242a0a529341503cc96ea (127.16.43.129:39971)
W20260620 09:26:25.065608 21369 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:26:25.065845 21369 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:26:25.065884 21369 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:26:25.067122 21369 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:26:25.067186 21369 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:26:25.067230 21369 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.43.130
I20260620 09:26:25.068641 21369 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:44371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781947562949448-16558-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781947562949448-16558-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.43.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781947562949448-16558-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.43.130
--webserver_port=0
--tserver_master_addrs=127.16.43.190:39727
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21369
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.43.130
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781947562949448-16558-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:26:25.068929 21369 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:26:25.069275 21369 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:26:25.074687 21374 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:26:25.074790 21369 server_base.cc:1034] running on GCE node
W20260620 09:26:25.074606 21375 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:26:25.074687 21377 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:26:25.075111 21369 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:26:25.075362 21369 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:26:25.076532 21369 hybrid_clock.cc:648] HybridClock initialized: now 1781947585076523 us; error 38 us; skew 500 ppm
I20260620 09:26:25.077828 21369 webserver.cc:460] Webserver started at http://127.16.43.130:40905/ using document root <none> and password file <none>
I20260620 09:26:25.078088 21369 fs_manager.cc:359] Metadata directory not provided
I20260620 09:26:25.078171 21369 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:26:25.078329 21369 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:26:25.079403 21369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781947562949448-16558-0/minicluster-data/ts-1/data/instance:
uuid: "a1533f26515b457f9add4c49aef29ff9"
format_stamp: "Formatted at 2026-06-20 09:26:25 on dist-test-slave-706q"
server_key: "e7a2f350ad7bd9a29915079527bc8df4"
server_key_iv: "9aad8ddf9617ebaed33c13d497eef219"
server_key_version: "encryptionkey@0"
I20260620 09:26:25.079828 21369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781947562949448-16558-0/minicluster-data/ts-1/wal/instance:
uuid: "a1533f26515b457f9add4c49aef29ff9"
format_stamp: "Formatted at 2026-06-20 09:26:25 on dist-test-slave-706q"
server_key: "e7a2f350ad7bd9a29915079527bc8df4"
server_key_iv: "9aad8ddf9617ebaed33c13d497eef219"
server_key_version: "encryptionkey@0"
I20260620 09:26:25.081199 21369 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:26:25.081990 21383 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:25.082207 21369 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:26:25.082306 21369 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781947562949448-16558-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781947562949448-16558-0/minicluster-data/ts-1/wal
uuid: "a1533f26515b457f9add4c49aef29ff9"
format_stamp: "Formatted at 2026-06-20 09:26:25 on dist-test-slave-706q"
server_key: "e7a2f350ad7bd9a29915079527bc8df4"
server_key_iv: "9aad8ddf9617ebaed33c13d497eef219"
server_key_version: "encryptionkey@0"
I20260620 09:26:25.082393 21369 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781947562949448-16558-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781947562949448-16558-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781947562949448-16558-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:26:25.085788 21369 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:26:25.086040 21369 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:26:25.086185 21369 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:26:25.086391 21369 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:26:25.086688 21369 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:26:25.086742 21369 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:25.086781 21369 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:26:25.086809 21369 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:25.092343 21369 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.130:40837
I20260620 09:26:25.092433 21496 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.130:40837 every 8 connection(s)
I20260620 09:26:25.092738 21369 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781947562949448-16558-0/minicluster-data/ts-1/data/info.pb
I20260620 09:26:25.095183 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 21369
I20260620 09:26:25.095309 16558 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781947562949448-16558-0/minicluster-data/ts-1/wal/instance
I20260620 09:26:25.095498 16558 external_mini_cluster.cc:1428] Setting key cd88d97a8751f388b33f2dbf0d96a7de
I20260620 09:26:25.096026 21236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:25.093512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.130:37027 (local address 127.16.43.190:39727)
0620 09:26:25.093595 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:25.093598 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:25.093759 (+ 161us) server_negotiation.cc:407] Connection header received
0620 09:26:25.094197 (+ 438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:25.094197 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:25.094203 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:25.094250 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:25.095683 (+ 1433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:25.095685 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:25.095685 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:26:25.095691 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:25.095713 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:26:25.095720 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:25.095795 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:25.095945 (+ 150us) server_negotiation.cc:299] Negotiation successful
0620 09:26:25.095960 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 09:26:25.096268 21499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:25.093445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:39727 (local address 127.16.43.130:37027)
0620 09:26:25.093643 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:25.093675 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:25.094116 (+ 441us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:25.094311 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:25.094314 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:25.094322 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:25.095403 (+ 1081us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 09:26:25.095404 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:26:25.095445 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 09:26:25.095482 (+ 37us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:26:25.095832 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:26:25.095839 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:26:25.095843 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 09:26:25.095945 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 09:26:25.096017 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":62,"threads_started":1}
I20260620 09:26:25.096777 21497 heartbeater.cc:347] Connected to a master server at 127.16.43.190:39727
I20260620 09:26:25.096825 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781947562949448-16558-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781947562949448-16558-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781947562949448-16558-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781947562949448-16558-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.43.131:0
--local_ip_for_outbound_sockets=127.16.43.131
--webserver_interface=127.16.43.131
--webserver_port=0
--tserver_master_addrs=127.16.43.190:39727
--builtin_ntp_servers=127.16.43.148:44371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260620 09:26:25.096886 21497 heartbeater.cc:464] Registering TS with master...
I20260620 09:26:25.097043 21497 heartbeater.cc:511] Master 127.16.43.190:39727 requested a full tablet report, sending...
I20260620 09:26:25.097378 21178 ts_manager.cc:194] Registered new tserver with Master: a1533f26515b457f9add4c49aef29ff9 (127.16.43.130:40837)
W20260620 09:26:25.178355 21500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:26:25.178606 21500 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:26:25.178647 21500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:26:25.179817 21500 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:26:25.179886 21500 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:26:25.179930 21500 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.43.131
I20260620 09:26:25.181272 21500 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:44371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781947562949448-16558-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781947562949448-16558-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.43.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781947562949448-16558-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.43.131
--webserver_port=0
--tserver_master_addrs=127.16.43.190:39727
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21500
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.43.131
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781947562949448-16558-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:26:25.181543 21500 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:26:25.181851 21500 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 09:26:25.187114 21500 server_base.cc:1034] running on GCE node
W20260620 09:26:25.187096 21506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:26:25.187084 21508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:26:25.187131 21505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:26:25.187638 21500 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:26:25.187898 21500 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:26:25.189075 21500 hybrid_clock.cc:648] HybridClock initialized: now 1781947585189067 us; error 41 us; skew 500 ppm
I20260620 09:26:25.190352 21500 webserver.cc:460] Webserver started at http://127.16.43.131:42607/ using document root <none> and password file <none>
I20260620 09:26:25.190658 21500 fs_manager.cc:359] Metadata directory not provided
I20260620 09:26:25.190713 21500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:26:25.190891 21500 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:26:25.192039 21500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781947562949448-16558-0/minicluster-data/ts-2/data/instance:
uuid: "c7e9214bdb9f4370b733cc9a3336b349"
format_stamp: "Formatted at 2026-06-20 09:26:25 on dist-test-slave-706q"
server_key: "edf73ce970b9f3fe458471ca5f8cffb4"
server_key_iv: "e0ebd55b2b3cf5d2d10a25579f388471"
server_key_version: "encryptionkey@0"
I20260620 09:26:25.192401 21500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781947562949448-16558-0/minicluster-data/ts-2/wal/instance:
uuid: "c7e9214bdb9f4370b733cc9a3336b349"
format_stamp: "Formatted at 2026-06-20 09:26:25 on dist-test-slave-706q"
server_key: "edf73ce970b9f3fe458471ca5f8cffb4"
server_key_iv: "e0ebd55b2b3cf5d2d10a25579f388471"
server_key_version: "encryptionkey@0"
I20260620 09:26:25.193758 21500 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:26:25.194684 21514 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:25.194870 21500 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:26:25.194988 21500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781947562949448-16558-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781947562949448-16558-0/minicluster-data/ts-2/wal
uuid: "c7e9214bdb9f4370b733cc9a3336b349"
format_stamp: "Formatted at 2026-06-20 09:26:25 on dist-test-slave-706q"
server_key: "edf73ce970b9f3fe458471ca5f8cffb4"
server_key_iv: "e0ebd55b2b3cf5d2d10a25579f388471"
server_key_version: "encryptionkey@0"
I20260620 09:26:25.195075 21500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781947562949448-16558-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781947562949448-16558-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781947562949448-16558-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:26:25.198982 21500 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:26:25.199275 21500 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:26:25.199456 21500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:26:25.199721 21500 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:26:25.200057 21500 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:26:25.200119 21500 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:25.200162 21500 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:26:25.200191 21500 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:25.206702 21500 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.131:36679
I20260620 09:26:25.206748 21627 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.131:36679 every 8 connection(s)
I20260620 09:26:25.207093 21500 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781947562949448-16558-0/minicluster-data/ts-2/data/info.pb
I20260620 09:26:25.210343 21236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:25.207911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.131:45645 (local address 127.16.43.190:39727)
0620 09:26:25.207956 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:25.207958 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:25.208217 (+ 259us) server_negotiation.cc:407] Connection header received
0620 09:26:25.208740 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:25.208742 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:25.208748 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:25.208806 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:25.210060 (+ 1254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:25.210062 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:25.210062 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:26:25.210067 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:25.210083 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:26:25.210087 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:25.210175 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:25.210257 (+ 82us) server_negotiation.cc:299] Negotiation successful
0620 09:26:25.210274 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 09:26:25.210549 21630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:25.207866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:39727 (local address 127.16.43.131:45645)
0620 09:26:25.208094 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:25.208127 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:25.208610 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:25.208888 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:25.208892 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:25.208899 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:25.209903 (+ 1004us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 09:26:25.209904 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:26:25.209931 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 09:26:25.209959 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:26:25.210176 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:26:25.210182 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:26:25.210185 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 09:26:25.210265 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 09:26:25.210327 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":98,"threads_started":1}
I20260620 09:26:25.211058 21628 heartbeater.cc:347] Connected to a master server at 127.16.43.190:39727
I20260620 09:26:25.211129 21628 heartbeater.cc:464] Registering TS with master...
I20260620 09:26:25.211293 21628 heartbeater.cc:511] Master 127.16.43.190:39727 requested a full tablet report, sending...
I20260620 09:26:25.211779 21178 ts_manager.cc:194] Registered new tserver with Master: c7e9214bdb9f4370b733cc9a3336b349 (127.16.43.131:36679)
I20260620 09:26:25.211930 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 21500
I20260620 09:26:25.212056 16558 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781947562949448-16558-0/minicluster-data/ts-2/wal/instance
I20260620 09:26:25.212251 16558 external_mini_cluster.cc:1428] Setting key c7dd16c35a93d9d46fae5be075a6d59e
I20260620 09:26:25.213022 16558 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 09:26:25.215593 21236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:25.214401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38284 (local address 127.16.43.190:39727)
0620 09:26:25.214446 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:25.214448 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:25.214587 (+ 139us) server_negotiation.cc:407] Connection header received
0620 09:26:25.214629 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:25.214630 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:25.214634 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:25.214682 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:25.215249 (+ 567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:25.215266 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:25.215266 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:26:25.215270 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:25.215288 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:26:25.215293 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:25.215350 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:25.215480 (+ 130us) server_negotiation.cc:299] Negotiation successful
0620 09:26:25.215492 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 09:26:25.217056 21178 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38284:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 09:26:25.218612 21178 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 09:26:25.222393 21636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:25.220788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.131:36679 (local address 127.0.0.1:56160)
0620 09:26:25.220911 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:25.220949 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:25.220997 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:25.221296 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:25.221303 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:25.221311 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:25.221893 (+ 582us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 09:26:25.221893 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:26:25.221910 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 09:26:25.221935 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:26:25.222225 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:26:25.222231 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:26:25.222235 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 09:26:25.222303 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 09:26:25.222316 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":80,"threads_started":1}
I20260620 09:26:25.222479 21640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:25.221313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33258 (local address 127.16.43.129:39971)
0620 09:26:25.221496 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:25.221501 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:25.221511 (+ 10us) server_negotiation.cc:407] Connection header received
0620 09:26:25.221535 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:25.221537 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:25.221547 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:25.221805 (+ 258us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:25.222153 (+ 348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:25.222157 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:25.222158 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:26:25.222161 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:25.222190 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:26:25.222207 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:25.222290 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:25.222373 (+ 83us) server_negotiation.cc:299] Negotiation successful
0620 09:26:25.222387 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":104,"threads_started":1}
I20260620 09:26:25.222506 21639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:25.221179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.129:39971 (local address 127.0.0.1:33258)
0620 09:26:25.221298 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:25.221307 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:25.221340 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:25.221651 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:25.221652 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:25.221655 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:25.222059 (+ 404us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 09:26:25.222060 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:26:25.222068 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 09:26:25.222080 (+ 12us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:26:25.222295 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:26:25.222296 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:26:25.222296 (+ 0us) client_negotiation.cc:769] Sending connection context
0620 09:26:25.222365 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 09:26:25.222380 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":65,"threads_started":1}
I20260620 09:26:25.222700 21637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:25.220886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56160 (local address 127.16.43.131:36679)
0620 09:26:25.221093 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:25.221102 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:25.221114 (+ 12us) server_negotiation.cc:407] Connection header received
0620 09:26:25.221148 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:25.221149 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:25.221158 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:25.221530 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:25.222088 (+ 558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:25.222092 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:25.222092 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:26:25.222096 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:25.222133 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:26:25.222151 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:25.222233 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:25.222597 (+ 364us) server_negotiation.cc:299] Negotiation successful
0620 09:26:25.222623 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":126,"threads_started":1}
I20260620 09:26:25.223408 21562 tablet_service.cc:1467] Processing CreateTablet for tablet 211dff6355144e3da514e867946c28a8 (DEFAULT_TABLE table=test-table [id=25c11426f1cd4bd7892bb8f3b68425e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:26:25.223407 21300 tablet_service.cc:1467] Processing CreateTablet for tablet 211dff6355144e3da514e867946c28a8 (DEFAULT_TABLE table=test-table [id=25c11426f1cd4bd7892bb8f3b68425e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:26:25.223816 21562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 211dff6355144e3da514e867946c28a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:26:25.223816 21300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 211dff6355144e3da514e867946c28a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:26:25.224210 21638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:25.221003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.130:40837 (local address 127.0.0.1:47174)
0620 09:26:25.221129 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:25.221138 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:25.221172 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:25.222301 (+ 1129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:25.222302 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:25.222306 (+ 4us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:25.223650 (+ 1344us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 09:26:25.223651 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:26:25.223657 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 09:26:25.223669 (+ 12us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:26:25.224069 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:26:25.224070 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:26:25.224071 (+ 1us) client_negotiation.cc:769] Sending connection context
0620 09:26:25.224133 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 09:26:25.224146 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":77,"threads_started":1}
I20260620 09:26:25.224253 21641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:25.221408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47174 (local address 127.16.43.130:40837)
0620 09:26:25.222135 (+ 727us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:25.222139 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:25.222150 (+ 11us) server_negotiation.cc:407] Connection header received
0620 09:26:25.222177 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:25.222179 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:25.222190 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:25.222927 (+ 737us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:25.223941 (+ 1014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:25.223944 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:25.223944 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:26:25.223948 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:25.223974 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:26:25.223985 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:25.224074 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:25.224149 (+ 75us) server_negotiation.cc:299] Negotiation successful
0620 09:26:25.224164 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":685,"thread_start_us":88,"threads_started":1}
I20260620 09:26:25.224838 21431 tablet_service.cc:1467] Processing CreateTablet for tablet 211dff6355144e3da514e867946c28a8 (DEFAULT_TABLE table=test-table [id=25c11426f1cd4bd7892bb8f3b68425e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:26:25.225165 21431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 211dff6355144e3da514e867946c28a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:26:25.227293 21642 tablet_bootstrap.cc:492] T 211dff6355144e3da514e867946c28a8 P c7e9214bdb9f4370b733cc9a3336b349: Bootstrap starting.
I20260620 09:26:25.227981 21644 tablet_bootstrap.cc:492] T 211dff6355144e3da514e867946c28a8 P a1533f26515b457f9add4c49aef29ff9: Bootstrap starting.
I20260620 09:26:25.228032 21642 tablet_bootstrap.cc:654] T 211dff6355144e3da514e867946c28a8 P c7e9214bdb9f4370b733cc9a3336b349: Neither blocks nor log segments found. Creating new log.
I20260620 09:26:25.228366 21642 log.cc:824] T 211dff6355144e3da514e867946c28a8 P c7e9214bdb9f4370b733cc9a3336b349: Log is configured to *not* fsync() on all Append() calls
I20260620 09:26:25.228587 21644 tablet_bootstrap.cc:654] T 211dff6355144e3da514e867946c28a8 P a1533f26515b457f9add4c49aef29ff9: Neither blocks nor log segments found. Creating new log.
I20260620 09:26:25.228753 21643 tablet_bootstrap.cc:492] T 211dff6355144e3da514e867946c28a8 P 3c77cefff54242a0a529341503cc96ea: Bootstrap starting.
I20260620 09:26:25.228914 21644 log.cc:824] T 211dff6355144e3da514e867946c28a8 P a1533f26515b457f9add4c49aef29ff9: Log is configured to *not* fsync() on all Append() calls
I20260620 09:26:25.229126 21642 tablet_bootstrap.cc:492] T 211dff6355144e3da514e867946c28a8 P c7e9214bdb9f4370b733cc9a3336b349: No bootstrap required, opened a new log
I20260620 09:26:25.229205 21642 ts_tablet_manager.cc:1397] T 211dff6355144e3da514e867946c28a8 P c7e9214bdb9f4370b733cc9a3336b349: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 09:26:25.229597 21643 tablet_bootstrap.cc:654] T 211dff6355144e3da514e867946c28a8 P 3c77cefff54242a0a529341503cc96ea: Neither blocks nor log segments found. Creating new log.
I20260620 09:26:25.229903 21643 log.cc:824] T 211dff6355144e3da514e867946c28a8 P 3c77cefff54242a0a529341503cc96ea: Log is configured to *not* fsync() on all Append() calls
I20260620 09:26:25.230674 21643 tablet_bootstrap.cc:492] T 211dff6355144e3da514e867946c28a8 P 3c77cefff54242a0a529341503cc96ea: No bootstrap required, opened a new log
I20260620 09:26:25.230674 21644 tablet_bootstrap.cc:492] T 211dff6355144e3da514e867946c28a8 P a1533f26515b457f9add4c49aef29ff9: No bootstrap required, opened a new log
I20260620 09:26:25.230738 21643 ts_tablet_manager.cc:1397] T 211dff6355144e3da514e867946c28a8 P 3c77cefff54242a0a529341503cc96ea: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 09:26:25.230738 21644 ts_tablet_manager.cc:1397] T 211dff6355144e3da514e867946c28a8 P a1533f26515b457f9add4c49aef29ff9: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260620 09:26:25.231510 21642 raft_consensus.cc:348] T 211dff6355144e3da514e867946c28a8 P c7e9214bdb9f4370b733cc9a3336b349 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7e9214bdb9f4370b733cc9a3336b349" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 36679 } } peers { permanent_uuid: "a1533f26515b457f9add4c49aef29ff9" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 40837 } } peers { permanent_uuid: "3c77cefff54242a0a529341503cc96ea" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 39971 } }
I20260620 09:26:25.231763 21642 raft_consensus.cc:374] T 211dff6355144e3da514e867946c28a8 P c7e9214bdb9f4370b733cc9a3336b349 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:26:25.231809 21642 raft_consensus.cc:729] T 211dff6355144e3da514e867946c28a8 P c7e9214bdb9f4370b733cc9a3336b349 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7e9214bdb9f4370b733cc9a3336b349, State: Initialized, Role: FOLLOWER
I20260620 09:26:25.231930 21642 consensus_queue.cc:260] T 211dff6355144e3da514e867946c28a8 P c7e9214bdb9f4370b733cc9a3336b349 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7e9214bdb9f4370b733cc9a3336b349" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 36679 } } peers { permanent_uuid: "a1533f26515b457f9add4c49aef29ff9" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 40837 } } peers { permanent_uuid: "3c77cefff54242a0a529341503cc96ea" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 39971 } }
I20260620 09:26:25.232236 21642 ts_tablet_manager.cc:1428] T 211dff6355144e3da514e867946c28a8 P c7e9214bdb9f4370b733cc9a3336b349: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260620 09:26:25.232357 21628 heartbeater.cc:503] Master 127.16.43.190:39727 was elected leader, sending a full tablet report...
I20260620 09:26:25.233403 21643 raft_consensus.cc:348] T 211dff6355144e3da514e867946c28a8 P 3c77cefff54242a0a529341503cc96ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7e9214bdb9f4370b733cc9a3336b349" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 36679 } } peers { permanent_uuid: "a1533f26515b457f9add4c49aef29ff9" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 40837 } } peers { permanent_uuid: "3c77cefff54242a0a529341503cc96ea" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 39971 } }
I20260620 09:26:25.233403 21644 raft_consensus.cc:348] T 211dff6355144e3da514e867946c28a8 P a1533f26515b457f9add4c49aef29ff9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7e9214bdb9f4370b733cc9a3336b349" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 36679 } } peers { permanent_uuid: "a1533f26515b457f9add4c49aef29ff9" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 40837 } } peers { permanent_uuid: "3c77cefff54242a0a529341503cc96ea" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 39971 } }
I20260620 09:26:25.233562 21643 raft_consensus.cc:374] T 211dff6355144e3da514e867946c28a8 P 3c77cefff54242a0a529341503cc96ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:26:25.233562 21644 raft_consensus.cc:374] T 211dff6355144e3da514e867946c28a8 P a1533f26515b457f9add4c49aef29ff9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:26:25.233597 21643 raft_consensus.cc:729] T 211dff6355144e3da514e867946c28a8 P 3c77cefff54242a0a529341503cc96ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c77cefff54242a0a529341503cc96ea, State: Initialized, Role: FOLLOWER
I20260620 09:26:25.233597 21644 raft_consensus.cc:729] T 211dff6355144e3da514e867946c28a8 P a1533f26515b457f9add4c49aef29ff9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1533f26515b457f9add4c49aef29ff9, State: Initialized, Role: FOLLOWER
I20260620 09:26:25.233727 21643 consensus_queue.cc:260] T 211dff6355144e3da514e867946c28a8 P 3c77cefff54242a0a529341503cc96ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7e9214bdb9f4370b733cc9a3336b349" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 36679 } } peers { permanent_uuid: "a1533f26515b457f9add4c49aef29ff9" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 40837 } } peers { permanent_uuid: "3c77cefff54242a0a529341503cc96ea" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 39971 } }
I20260620 09:26:25.233717 21644 consensus_queue.cc:260] T 211dff6355144e3da514e867946c28a8 P a1533f26515b457f9add4c49aef29ff9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7e9214bdb9f4370b733cc9a3336b349" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 36679 } } peers { permanent_uuid: "a1533f26515b457f9add4c49aef29ff9" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 40837 } } peers { permanent_uuid: "3c77cefff54242a0a529341503cc96ea" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 39971 } }
I20260620 09:26:25.233978 21644 ts_tablet_manager.cc:1428] T 211dff6355144e3da514e867946c28a8 P a1533f26515b457f9add4c49aef29ff9: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 09:26:25.233978 21643 ts_tablet_manager.cc:1428] T 211dff6355144e3da514e867946c28a8 P 3c77cefff54242a0a529341503cc96ea: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 09:26:25.234014 21497 heartbeater.cc:503] Master 127.16.43.190:39727 was elected leader, sending a full tablet report...
I20260620 09:26:25.234014 21366 heartbeater.cc:503] Master 127.16.43.190:39727 was elected leader, sending a full tablet report...
I20260620 09:26:25.253334 21649 raft_consensus.cc:482] T 211dff6355144e3da514e867946c28a8 P 3c77cefff54242a0a529341503cc96ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:26:25.253490 21649 raft_consensus.cc:504] T 211dff6355144e3da514e867946c28a8 P 3c77cefff54242a0a529341503cc96ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7e9214bdb9f4370b733cc9a3336b349" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 36679 } } peers { permanent_uuid: "a1533f26515b457f9add4c49aef29ff9" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 40837 } } peers { permanent_uuid: "3c77cefff54242a0a529341503cc96ea" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 39971 } }
I20260620 09:26:25.253885 21649 leader_election.cc:288] T 211dff6355144e3da514e867946c28a8 P 3c77cefff54242a0a529341503cc96ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c7e9214bdb9f4370b733cc9a3336b349 (127.16.43.131:36679), a1533f26515b457f9add4c49aef29ff9 (127.16.43.130:40837)
I20260620 09:26:25.255314 21368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:25.254090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.131:36679 (local address 127.16.43.129:40737)
0620 09:26:25.254159 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:25.254172 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:25.254211 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:25.254446 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:25.254447 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:25.254450 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:25.254976 (+ 526us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 09:26:25.254977 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:26:25.254984 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 09:26:25.254995 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:26:25.255186 (+ 191us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:26:25.255187 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:26:25.255188 (+ 1us) client_negotiation.cc:769] Sending connection context
0620 09:26:25.255231 (+ 43us) client_negotiation.cc:240] Negotiation successful
0620 09:26:25.255247 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260620 09:26:25.255318 21637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:25.254164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.129:40737 (local address 127.16.43.131:36679)
0620 09:26:25.254230 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:25.254232 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:25.254269 (+ 37us) server_negotiation.cc:407] Connection header received
0620 09:26:25.254297 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:25.254298 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:25.254305 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:25.254416 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:25.255079 (+ 663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:25.255081 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:25.255081 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:26:25.255086 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:25.255102 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:26:25.255108 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:25.255192 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:25.255245 (+ 53us) server_negotiation.cc:299] Negotiation successful
0620 09:26:25.255259 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 09:26:25.255513 21641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:25.254333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.129:52257 (local address 127.16.43.130:40837)
0620 09:26:25.254401 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:25.254403 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:25.254434 (+ 31us) server_negotiation.cc:407] Connection header received
0620 09:26:25.254506 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:25.254507 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:25.254512 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:25.254572 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:25.255263 (+ 691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:25.255264 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:25.255265 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:26:25.255270 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:25.255285 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:26:25.255289 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:25.255341 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:25.255417 (+ 76us) server_negotiation.cc:299] Negotiation successful
0620 09:26:25.255433 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 09:26:25.255694 21651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:25.254209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.130:40837 (local address 127.16.43.129:52257)
0620 09:26:25.254372 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:25.254382 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:25.254440 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:25.254649 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:25.254650 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:25.254659 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:25.255129 (+ 470us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 09:26:25.255129 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:26:25.255137 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 09:26:25.255149 (+ 12us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:26:25.255337 (+ 188us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:26:25.255338 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:26:25.255338 (+ 0us) client_negotiation.cc:769] Sending connection context
0620 09:26:25.255615 (+ 277us) client_negotiation.cc:240] Negotiation successful
0620 09:26:25.255628 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":107,"threads_started":1}
I20260620 09:26:25.255679 21582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "211dff6355144e3da514e867946c28a8" candidate_uuid: "3c77cefff54242a0a529341503cc96ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7e9214bdb9f4370b733cc9a3336b349" is_pre_election: true
I20260620 09:26:25.255859 21582 raft_consensus.cc:2454] T 211dff6355144e3da514e867946c28a8 P c7e9214bdb9f4370b733cc9a3336b349 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c77cefff54242a0a529341503cc96ea in term 0.
I20260620 09:26:25.256038 21451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "211dff6355144e3da514e867946c28a8" candidate_uuid: "3c77cefff54242a0a529341503cc96ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1533f26515b457f9add4c49aef29ff9" is_pre_election: true
I20260620 09:26:25.256125 21253 leader_election.cc:302] T 211dff6355144e3da514e867946c28a8 P 3c77cefff54242a0a529341503cc96ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c77cefff54242a0a529341503cc96ea, c7e9214bdb9f4370b733cc9a3336b349; no voters:
I20260620 09:26:25.256187 21451 raft_consensus.cc:2454] T 211dff6355144e3da514e867946c28a8 P a1533f26515b457f9add4c49aef29ff9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c77cefff54242a0a529341503cc96ea in term 0.
I20260620 09:26:25.256250 21649 raft_consensus.cc:2781] T 211dff6355144e3da514e867946c28a8 P 3c77cefff54242a0a529341503cc96ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 09:26:25.256299 21649 raft_consensus.cc:482] T 211dff6355144e3da514e867946c28a8 P 3c77cefff54242a0a529341503cc96ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 09:26:25.256325 21649 raft_consensus.cc:3037] T 211dff6355144e3da514e867946c28a8 P 3c77cefff54242a0a529341503cc96ea [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:26:25.257059 21649 raft_consensus.cc:504] T 211dff6355144e3da514e867946c28a8 P 3c77cefff54242a0a529341503cc96ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7e9214bdb9f4370b733cc9a3336b349" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 36679 } } peers { permanent_uuid: "a1533f26515b457f9add4c49aef29ff9" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 40837 } } peers { permanent_uuid: "3c77cefff54242a0a529341503cc96ea" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 39971 } }
I20260620 09:26:25.257200 21649 leader_election.cc:288] T 211dff6355144e3da514e867946c28a8 P 3c77cefff54242a0a529341503cc96ea [CANDIDATE]: Term 1 election: Requested vote from peers c7e9214bdb9f4370b733cc9a3336b349 (127.16.43.131:36679), a1533f26515b457f9add4c49aef29ff9 (127.16.43.130:40837)
I20260620 09:26:25.257380 21451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "211dff6355144e3da514e867946c28a8" candidate_uuid: "3c77cefff54242a0a529341503cc96ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1533f26515b457f9add4c49aef29ff9"
I20260620 09:26:25.257411 21582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "211dff6355144e3da514e867946c28a8" candidate_uuid: "3c77cefff54242a0a529341503cc96ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7e9214bdb9f4370b733cc9a3336b349"
I20260620 09:26:25.257479 21451 raft_consensus.cc:3037] T 211dff6355144e3da514e867946c28a8 P a1533f26515b457f9add4c49aef29ff9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:26:25.257475 21582 raft_consensus.cc:3037] T 211dff6355144e3da514e867946c28a8 P c7e9214bdb9f4370b733cc9a3336b349 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:26:25.258189 21582 raft_consensus.cc:2454] T 211dff6355144e3da514e867946c28a8 P c7e9214bdb9f4370b733cc9a3336b349 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c77cefff54242a0a529341503cc96ea in term 1.
I20260620 09:26:25.258215 21451 raft_consensus.cc:2454] T 211dff6355144e3da514e867946c28a8 P a1533f26515b457f9add4c49aef29ff9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c77cefff54242a0a529341503cc96ea in term 1.
I20260620 09:26:25.258363 21253 leader_election.cc:302] T 211dff6355144e3da514e867946c28a8 P 3c77cefff54242a0a529341503cc96ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c77cefff54242a0a529341503cc96ea, a1533f26515b457f9add4c49aef29ff9; no voters:
I20260620 09:26:25.258466 21649 raft_consensus.cc:2781] T 211dff6355144e3da514e867946c28a8 P 3c77cefff54242a0a529341503cc96ea [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:26:25.258638 21649 raft_consensus.cc:686] T 211dff6355144e3da514e867946c28a8 P 3c77cefff54242a0a529341503cc96ea [term 1 LEADER]: Becoming Leader. State: Replica: 3c77cefff54242a0a529341503cc96ea, State: Running, Role: LEADER
I20260620 09:26:25.258755 21649 consensus_queue.cc:237] T 211dff6355144e3da514e867946c28a8 P 3c77cefff54242a0a529341503cc96ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7e9214bdb9f4370b733cc9a3336b349" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 36679 } } peers { permanent_uuid: "a1533f26515b457f9add4c49aef29ff9" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 40837 } } peers { permanent_uuid: "3c77cefff54242a0a529341503cc96ea" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 39971 } }
I20260620 09:26:25.259713 21177 catalog_manager.cc:5495] T 211dff6355144e3da514e867946c28a8 P 3c77cefff54242a0a529341503cc96ea reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c77cefff54242a0a529341503cc96ea (127.16.43.129). New cstate: current_term: 1 leader_uuid: "3c77cefff54242a0a529341503cc96ea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7e9214bdb9f4370b733cc9a3336b349" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 36679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1533f26515b457f9add4c49aef29ff9" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 40837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c77cefff54242a0a529341503cc96ea" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 39971 } health_report { overall_health: HEALTHY } } }
I20260620 09:26:25.268990 21640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:25.268153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33274 (local address 127.16.43.129:39971)
0620 09:26:25.268195 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:25.268197 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:25.268205 (+ 8us) server_negotiation.cc:407] Connection header received
0620 09:26:25.268246 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:25.268247 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:25.268257 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:25.268311 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:25.268826 (+ 515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:25.268827 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:25.268828 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:26:25.268832 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:25.268845 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:26:25.268848 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:25.268897 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:25.268928 (+ 31us) server_negotiation.cc:299] Negotiation successful
0620 09:26:25.268939 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 09:26:25.270974 21582 raft_consensus.cc:1261] T 211dff6355144e3da514e867946c28a8 P c7e9214bdb9f4370b733cc9a3336b349 [term 1 FOLLOWER]: Refusing update from remote peer 3c77cefff54242a0a529341503cc96ea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:26:25.270974 21451 raft_consensus.cc:1261] T 211dff6355144e3da514e867946c28a8 P a1533f26515b457f9add4c49aef29ff9 [term 1 FOLLOWER]: Refusing update from remote peer 3c77cefff54242a0a529341503cc96ea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:26:25.271369 21652 consensus_queue.cc:1035] T 211dff6355144e3da514e867946c28a8 P 3c77cefff54242a0a529341503cc96ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "c7e9214bdb9f4370b733cc9a3336b349" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:26:25.271560 21649 consensus_queue.cc:1035] T 211dff6355144e3da514e867946c28a8 P 3c77cefff54242a0a529341503cc96ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1533f26515b457f9add4c49aef29ff9" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 40837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:26:25.274058 21657 mvcc.cc:205] Tried to move back new op lower bound from 7298857309266575360 to 7298857309220556800. Current Snapshot: MvccSnapshot[applied={T|T < 7298857309266575360}]
I20260620 09:26:25.276811 21177 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:38284:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:26:25.276892 21177 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:38284:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:26:25.277917 21177 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cf8d77d815e34f50a03e113fa415cef4: Sending DeleteTablet for 3 replicas of tablet 211dff6355144e3da514e867946c28a8
I20260620 09:26:25.278362 21562 tablet_service.cc:1514] Processing DeleteTablet for tablet 211dff6355144e3da514e867946c28a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:26:25 UTC) from {username='slave'} at 127.0.0.1:56160
I20260620 09:26:25.278461 21431 tablet_service.cc:1514] Processing DeleteTablet for tablet 211dff6355144e3da514e867946c28a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:26:25 UTC) from {username='slave'} at 127.0.0.1:47174
I20260620 09:26:25.278721 21666 tablet_replica.cc:330] T 211dff6355144e3da514e867946c28a8 P a1533f26515b457f9add4c49aef29ff9: stopping tablet replica
I20260620 09:26:25.278760 21300 tablet_service.cc:1514] Processing DeleteTablet for tablet 211dff6355144e3da514e867946c28a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:26:25 UTC) from {username='slave'} at 127.0.0.1:33258
I20260620 09:26:25.278803 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 21238
I20260620 09:26:25.279366 21666 raft_consensus.cc:2229] T 211dff6355144e3da514e867946c28a8 P a1533f26515b457f9add4c49aef29ff9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:26:25.279515 21666 raft_consensus.cc:2258] T 211dff6355144e3da514e867946c28a8 P a1533f26515b457f9add4c49aef29ff9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:26:25.280054 21666 ts_tablet_manager.cc:1905] T 211dff6355144e3da514e867946c28a8 P a1533f26515b457f9add4c49aef29ff9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:26:25.280148 21665 tablet_replica.cc:330] T 211dff6355144e3da514e867946c28a8 P c7e9214bdb9f4370b733cc9a3336b349: stopping tablet replica
I20260620 09:26:25.280272 21665 raft_consensus.cc:2229] T 211dff6355144e3da514e867946c28a8 P c7e9214bdb9f4370b733cc9a3336b349 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:26:25.280388 21665 raft_consensus.cc:2258] T 211dff6355144e3da514e867946c28a8 P c7e9214bdb9f4370b733cc9a3336b349 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:26:25.280861 21665 ts_tablet_manager.cc:1905] T 211dff6355144e3da514e867946c28a8 P c7e9214bdb9f4370b733cc9a3336b349: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:26:25.281620 21666 ts_tablet_manager.cc:1918] T 211dff6355144e3da514e867946c28a8 P a1533f26515b457f9add4c49aef29ff9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:26:25.281714 21666 log.cc:1196] T 211dff6355144e3da514e867946c28a8 P a1533f26515b457f9add4c49aef29ff9: Deleting WAL directory at /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781947562949448-16558-0/minicluster-data/ts-1/wal/wals/211dff6355144e3da514e867946c28a8
I20260620 09:26:25.281950 21666 ts_tablet_manager.cc:1939] T 211dff6355144e3da514e867946c28a8 P a1533f26515b457f9add4c49aef29ff9: Deleting consensus metadata
I20260620 09:26:25.282368 21162 catalog_manager.cc:4841] TS a1533f26515b457f9add4c49aef29ff9 (127.16.43.130:40837): tablet 211dff6355144e3da514e867946c28a8 (table test-table [id=25c11426f1cd4bd7892bb8f3b68425e8]) successfully deleted
I20260620 09:26:25.283030 21665 ts_tablet_manager.cc:1918] T 211dff6355144e3da514e867946c28a8 P c7e9214bdb9f4370b733cc9a3336b349: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:26:25.283124 21665 log.cc:1196] T 211dff6355144e3da514e867946c28a8 P c7e9214bdb9f4370b733cc9a3336b349: Deleting WAL directory at /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781947562949448-16558-0/minicluster-data/ts-2/wal/wals/211dff6355144e3da514e867946c28a8
I20260620 09:26:25.283401 21665 ts_tablet_manager.cc:1939] T 211dff6355144e3da514e867946c28a8 P c7e9214bdb9f4370b733cc9a3336b349: Deleting consensus metadata
I20260620 09:26:25.283872 21162 catalog_manager.cc:4841] TS c7e9214bdb9f4370b733cc9a3336b349 (127.16.43.131:36679): tablet 211dff6355144e3da514e867946c28a8 (table test-table [id=25c11426f1cd4bd7892bb8f3b68425e8]) successfully deleted
W20260620 09:26:25.286039 21162 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.43.129:39971 (error 108)
I20260620 09:26:25.286266 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 21369
I20260620 09:26:25.287281 21638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:25.287133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.129:39971 (local address 127.0.0.1:33278)
0620 09:26:25.287204 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:25.287251 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.129:39971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260620 09:26:25.287410 21162 catalog_manager.cc:4584] TS 3c77cefff54242a0a529341503cc96ea (127.16.43.129:39971): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 211dff6355144e3da514e867946c28a8: Network error: Client connection negotiation failed: client connection to 127.16.43.129:39971: connect: Connection refused (error 111)
I20260620 09:26:25.291952 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 21500
I20260620 09:26:25.297053 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 21147
2026-06-20T09:26:25Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (607 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260620 09:26:25.316953 16558 mini_oidc.cc:152] Starting JWKS server
I20260620 09:26:25.317986 16558 webserver.cc:460] Webserver started at https://127.0.0.1:45651/ using document root <none> and password file <none>
I20260620 09:26:25.318063 16558 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 09:26:25.318219 16558 webserver.cc:460] Webserver started at http://127.0.0.1:45743/ using document root <none> and password file <none>
2026-06-20T09:26:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:26:25Z Disabled control of system clock
I20260620 09:26:25.325955 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781947562949448-16558-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781947562949448-16558-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781947562949448-16558-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.43.190:45271
--webserver_interface=127.16.43.190
--webserver_port=0
--builtin_ntp_servers=127.16.43.148:37797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.43.190:45271
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45651/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781947562949448-16558-0/testchainca.cert with env {}
W20260620 09:26:25.408922 21673 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:26:25.409150 21673 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:26:25.409175 21673 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:26:25.410332 21673 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:26:25.410369 21673 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:26:25.410382 21673 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:26:25.410394 21673 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:26:25.410406 21673 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 09:26:25.410419 21673 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 09:26:25.410430 21673 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45651/jwks.json
I20260620 09:26:25.412115 21673 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:37797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781947562949448-16558-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781947562949448-16558-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.43.190:45271
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.43.190:45271
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45651/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.43.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781947562949448-16558-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21673
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781947562949448-16558-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:26:25.412335 21673 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:26:25.412498 21673 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:26:25.417582 21679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:26:25.417694 21673 server_base.cc:1034] running on GCE node
W20260620 09:26:25.417514 21678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:26:25.417497 21681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:26:25.418004 21673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:26:25.418277 21673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:26:25.419467 21673 hybrid_clock.cc:648] HybridClock initialized: now 1781947585419439 us; error 41 us; skew 500 ppm
I20260620 09:26:25.420951 21673 webserver.cc:460] Webserver started at http://127.16.43.190:35475/ using document root <none> and password file <none>
I20260620 09:26:25.421248 21673 fs_manager.cc:359] Metadata directory not provided
I20260620 09:26:25.421307 21673 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:26:25.421741 21673 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:26:25.422895 21673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781947562949448-16558-0/minicluster-data/master-0/data/instance:
uuid: "25983ce26c544311b68e6b34957e21b0"
format_stamp: "Formatted at 2026-06-20 09:26:25 on dist-test-slave-706q"
server_key: "950c8901df7e4a7a1c53182d999e6db1"
server_key_iv: "f39513d44cc44002cb6b677ce7b01398"
server_key_version: "encryptionkey@0"
I20260620 09:26:25.423274 21673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781947562949448-16558-0/minicluster-data/master-0/wal/instance:
uuid: "25983ce26c544311b68e6b34957e21b0"
format_stamp: "Formatted at 2026-06-20 09:26:25 on dist-test-slave-706q"
server_key: "950c8901df7e4a7a1c53182d999e6db1"
server_key_iv: "f39513d44cc44002cb6b677ce7b01398"
server_key_version: "encryptionkey@0"
I20260620 09:26:25.424700 21673 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.004s sys 0.000s
I20260620 09:26:25.425495 21687 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:25.425709 21673 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:26:25.425802 21673 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781947562949448-16558-0/minicluster-data/master-0/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781947562949448-16558-0/minicluster-data/master-0/wal
uuid: "25983ce26c544311b68e6b34957e21b0"
format_stamp: "Formatted at 2026-06-20 09:26:25 on dist-test-slave-706q"
server_key: "950c8901df7e4a7a1c53182d999e6db1"
server_key_iv: "f39513d44cc44002cb6b677ce7b01398"
server_key_version: "encryptionkey@0"
I20260620 09:26:25.425884 21673 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781947562949448-16558-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781947562949448-16558-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781947562949448-16558-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:26:25.466679 21673 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:26:25.467039 21673 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:26:25.467228 21673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:26:25.471149 21673 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.190:45271
I20260620 09:26:25.471207 21739 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.190:45271 every 8 connection(s)
I20260620 09:26:25.471608 21673 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
I20260620 09:26:25.472632 21740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:26:25.475051 21740 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25983ce26c544311b68e6b34957e21b0: Bootstrap starting.
I20260620 09:26:25.475765 21740 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 25983ce26c544311b68e6b34957e21b0: Neither blocks nor log segments found. Creating new log.
I20260620 09:26:25.476069 21740 log.cc:824] T 00000000000000000000000000000000 P 25983ce26c544311b68e6b34957e21b0: Log is configured to *not* fsync() on all Append() calls
I20260620 09:26:25.476847 21740 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25983ce26c544311b68e6b34957e21b0: No bootstrap required, opened a new log
I20260620 09:26:25.478482 21740 raft_consensus.cc:348] T 00000000000000000000000000000000 P 25983ce26c544311b68e6b34957e21b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25983ce26c544311b68e6b34957e21b0" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 45271 } }
I20260620 09:26:25.478631 21740 raft_consensus.cc:374] T 00000000000000000000000000000000 P 25983ce26c544311b68e6b34957e21b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:26:25.478663 21740 raft_consensus.cc:729] T 00000000000000000000000000000000 P 25983ce26c544311b68e6b34957e21b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25983ce26c544311b68e6b34957e21b0, State: Initialized, Role: FOLLOWER
I20260620 09:26:25.478821 21740 consensus_queue.cc:260] T 00000000000000000000000000000000 P 25983ce26c544311b68e6b34957e21b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25983ce26c544311b68e6b34957e21b0" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 45271 } }
I20260620 09:26:25.478896 21740 raft_consensus.cc:388] T 00000000000000000000000000000000 P 25983ce26c544311b68e6b34957e21b0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:26:25.478919 21740 raft_consensus.cc:482] T 00000000000000000000000000000000 P 25983ce26c544311b68e6b34957e21b0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:26:25.478966 21740 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 25983ce26c544311b68e6b34957e21b0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:26:25.479651 21740 raft_consensus.cc:504] T 00000000000000000000000000000000 P 25983ce26c544311b68e6b34957e21b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25983ce26c544311b68e6b34957e21b0" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 45271 } }
I20260620 09:26:25.479761 21740 leader_election.cc:302] T 00000000000000000000000000000000 P 25983ce26c544311b68e6b34957e21b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 25983ce26c544311b68e6b34957e21b0; no voters:
I20260620 09:26:25.479939 21740 leader_election.cc:288] T 00000000000000000000000000000000 P 25983ce26c544311b68e6b34957e21b0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:26:25.480005 21743 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 25983ce26c544311b68e6b34957e21b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:26:25.480185 21743 raft_consensus.cc:686] T 00000000000000000000000000000000 P 25983ce26c544311b68e6b34957e21b0 [term 1 LEADER]: Becoming Leader. State: Replica: 25983ce26c544311b68e6b34957e21b0, State: Running, Role: LEADER
I20260620 09:26:25.480225 21740 sys_catalog.cc:564] T 00000000000000000000000000000000 P 25983ce26c544311b68e6b34957e21b0 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:26:25.480294 21743 consensus_queue.cc:237] T 00000000000000000000000000000000 P 25983ce26c544311b68e6b34957e21b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25983ce26c544311b68e6b34957e21b0" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 45271 } }
I20260620 09:26:25.480701 21745 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25983ce26c544311b68e6b34957e21b0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 25983ce26c544311b68e6b34957e21b0. Latest consensus state: current_term: 1 leader_uuid: "25983ce26c544311b68e6b34957e21b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25983ce26c544311b68e6b34957e21b0" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 45271 } } }
I20260620 09:26:25.480795 21744 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25983ce26c544311b68e6b34957e21b0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "25983ce26c544311b68e6b34957e21b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25983ce26c544311b68e6b34957e21b0" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 45271 } } }
I20260620 09:26:25.480798 21745 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25983ce26c544311b68e6b34957e21b0 [sys.catalog]: This master's current role is: LEADER
I20260620 09:26:25.480872 21744 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25983ce26c544311b68e6b34957e21b0 [sys.catalog]: This master's current role is: LEADER
I20260620 09:26:25.481076 21752 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:26:25.481112 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 21673
I20260620 09:26:25.481243 16558 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781947562949448-16558-0/minicluster-data/master-0/wal/instance
I20260620 09:26:25.481449 16558 external_mini_cluster.cc:1428] Setting key bf26a32bf554605036793207b3b4479b
I20260620 09:26:25.481633 21752 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:26:25.483443 21752 catalog_manager.cc:1347] Generated new cluster ID: 126cbb66ee4b42d68dadcd0056eda5e6
I20260620 09:26:25.483493 21752 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:26:25.487735 21761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:25.481940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35606 (local address 127.16.43.190:45271)
0620 09:26:25.482137 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:25.482148 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:25.482351 (+ 203us) server_negotiation.cc:407] Connection header received
0620 09:26:25.482474 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:25.482481 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:25.482566 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:25.482737 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:25.482936 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:25.484347 (+ 1411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:25.485429 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:25.486123 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:25.486267 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:25.486978 (+ 711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:25.486983 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:25.486983 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:26:25.487204 (+ 221us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:25.487252 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:26:25.487267 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:25.487336 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:25.487404 (+ 68us) server_negotiation.cc:299] Negotiation successful
0620 09:26:25.487488 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":70,"threads_started":1}
I20260620 09:26:25.488618 16558 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 09:26:25.516953 21752 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:26:25.517549 21752 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:26:25.522426 21752 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 25983ce26c544311b68e6b34957e21b0: Generated new TSK 0
I20260620 09:26:25.522632 21752 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 09:26:25.551393 21770 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260620 09:26:25.551573 21770 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 09:26:25.551649 21761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:25.529393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35612 (local address 127.16.43.190:45271)
0620 09:26:25.529467 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:25.529469 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:25.529655 (+ 186us) server_negotiation.cc:407] Connection header received
0620 09:26:25.529759 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:25.529760 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:25.529769 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:25.529828 (+ 59us) server_negotiation.cc:226] Negotiated authn=JWT
0620 09:26:25.529995 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:25.530461 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:25.531617 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:25.531818 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:25.531875 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:25.532118 (+ 243us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 09:26:25.551374 (+ 19256us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45651/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 09:26:25.551572 (+ 198us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35612: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45651/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":50}
W20260620 09:26:25.551718 21761 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35612: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45651/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260620 09:26:25.551991 21769 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:26:25.529316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:45271 (local address 127.0.0.1:35612)
0620 09:26:25.529572 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:25.529585 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:25.529663 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:25.529852 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:25.529852 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:25.529855 (+ 3us) client_negotiation.cc:189] Negotiated authn=JWT
0620 09:26:25.529930 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:25.529932 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:25.530554 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:25.530555 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:25.531503 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:25.531507 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:25.531910 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:25.531911 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:25.531985 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:25.532008 (+ 23us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 09:26:25.551544 (+ 19536us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45651/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 09:26:25.551673 (+ 129us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.16.43.190:45271: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45651/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":147,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.16.43.190:45271: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45651/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260620 09:26:25.552476 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 21673
2026-06-20T09:26:25Z chronyd exiting
I20260620 09:26:25.588398 16558 test_util.cc:175] -----------------------------------------------
I20260620 09:26:25.588517 16558 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781947562949448-16558-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (272 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260620 09:26:25.589385 16558 mini_oidc.cc:152] Starting JWKS server
I20260620 09:26:25.590083 16558 webserver.cc:460] Webserver started at https://127.0.0.1:46143/ using document root <none> and password file <none>
I20260620 09:26:25.590153 16558 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 09:26:25.590324 16558 webserver.cc:460] Webserver started at http://127.0.0.1:41561/ using document root <none> and password file <none>
2026-06-20T09:26:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:26:25Z Disabled control of system clock
I20260620 09:26:25.597940 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781947562949448-16558-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781947562949448-16558-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781947562949448-16558-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.43.190:41299
--webserver_interface=127.16.43.190
--webserver_port=0
--builtin_ntp_servers=127.16.43.148:45287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.43.190:41299
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46143/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781947562949448-16558-0/testchainca.cert with env {}
W20260620 09:26:25.677613 21776 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:26:25.677913 21776 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:26:25.677958 21776 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:26:25.679280 21776 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:26:25.679366 21776 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:26:25.679388 21776 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:26:25.679409 21776 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:26:25.679435 21776 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 09:26:25.679467 21776 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 09:26:25.679488 21776 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46143/jwks.json
I20260620 09:26:25.681154 21776 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:45287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781947562949448-16558-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781947562949448-16558-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.43.190:41299
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.43.190:41299
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46143/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.43.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781947562949448-16558-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21776
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781947562949448-16558-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:26:25.681458 21776 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:26:25.681702 21776 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:26:25.686923 21781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:26:25.687093 21782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:26:25.686928 21784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:26:25.687103 21776 server_base.cc:1034] running on GCE node
I20260620 09:26:25.687458 21776 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:26:25.687839 21776 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:26:25.689035 21776 hybrid_clock.cc:648] HybridClock initialized: now 1781947585688998 us; error 47 us; skew 500 ppm
I20260620 09:26:25.690475 21776 webserver.cc:460] Webserver started at http://127.16.43.190:34231/ using document root <none> and password file <none>
I20260620 09:26:25.690775 21776 fs_manager.cc:359] Metadata directory not provided
I20260620 09:26:25.690852 21776 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:26:25.690999 21776 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:26:25.692293 21776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781947562949448-16558-0/minicluster-data/master-0/data/instance:
uuid: "0b1e0d9cb04f494e95365c1ed4e8bb01"
format_stamp: "Formatted at 2026-06-20 09:26:25 on dist-test-slave-706q"
server_key: "92acb861d2977b8958170287e7678b89"
server_key_iv: "ba8158ad138b6561290e9d200082724e"
server_key_version: "encryptionkey@0"
I20260620 09:26:25.692683 21776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781947562949448-16558-0/minicluster-data/master-0/wal/instance:
uuid: "0b1e0d9cb04f494e95365c1ed4e8bb01"
format_stamp: "Formatted at 2026-06-20 09:26:25 on dist-test-slave-706q"
server_key: "92acb861d2977b8958170287e7678b89"
server_key_iv: "ba8158ad138b6561290e9d200082724e"
server_key_version: "encryptionkey@0"
I20260620 09:26:25.694013 21776 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 09:26:25.694808 21790 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:25.694979 21776 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:26:25.695084 21776 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781947562949448-16558-0/minicluster-data/master-0/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781947562949448-16558-0/minicluster-data/master-0/wal
uuid: "0b1e0d9cb04f494e95365c1ed4e8bb01"
format_stamp: "Formatted at 2026-06-20 09:26:25 on dist-test-slave-706q"
server_key: "92acb861d2977b8958170287e7678b89"
server_key_iv: "ba8158ad138b6561290e9d200082724e"
server_key_version: "encryptionkey@0"
I20260620 09:26:25.695169 21776 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781947562949448-16558-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781947562949448-16558-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781947562949448-16558-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:26:25.711287 21776 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:26:25.711684 21776 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:26:25.711851 21776 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:26:25.716040 21776 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.190:41299
I20260620 09:26:25.716089 21842 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.190:41299 every 8 connection(s)
I20260620 09:26:25.716457 21776 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
I20260620 09:26:25.717398 21843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:26:25.719871 21843 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b1e0d9cb04f494e95365c1ed4e8bb01: Bootstrap starting.
I20260620 09:26:25.720454 21843 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0b1e0d9cb04f494e95365c1ed4e8bb01: Neither blocks nor log segments found. Creating new log.
I20260620 09:26:25.720748 21843 log.cc:824] T 00000000000000000000000000000000 P 0b1e0d9cb04f494e95365c1ed4e8bb01: Log is configured to *not* fsync() on all Append() calls
I20260620 09:26:25.721432 21843 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b1e0d9cb04f494e95365c1ed4e8bb01: No bootstrap required, opened a new log
I20260620 09:26:25.722690 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 21776
I20260620 09:26:25.722813 16558 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781947562949448-16558-0/minicluster-data/master-0/wal/instance
I20260620 09:26:25.722965 16558 external_mini_cluster.cc:1428] Setting key b886924bf8bd51a3723d28adcd4da1a3
I20260620 09:26:25.723217 21843 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0b1e0d9cb04f494e95365c1ed4e8bb01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b1e0d9cb04f494e95365c1ed4e8bb01" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 41299 } }
I20260620 09:26:25.723412 21843 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0b1e0d9cb04f494e95365c1ed4e8bb01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:26:25.723469 21843 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0b1e0d9cb04f494e95365c1ed4e8bb01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b1e0d9cb04f494e95365c1ed4e8bb01, State: Initialized, Role: FOLLOWER
I20260620 09:26:25.723625 21843 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b1e0d9cb04f494e95365c1ed4e8bb01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b1e0d9cb04f494e95365c1ed4e8bb01" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 41299 } }
I20260620 09:26:25.723752 21843 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0b1e0d9cb04f494e95365c1ed4e8bb01 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:26:25.723788 21843 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0b1e0d9cb04f494e95365c1ed4e8bb01 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:26:25.723837 21843 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0b1e0d9cb04f494e95365c1ed4e8bb01 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:26:25.724542 21843 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0b1e0d9cb04f494e95365c1ed4e8bb01 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b1e0d9cb04f494e95365c1ed4e8bb01" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 41299 } }
I20260620 09:26:25.724645 21843 leader_election.cc:302] T 00000000000000000000000000000000 P 0b1e0d9cb04f494e95365c1ed4e8bb01 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0b1e0d9cb04f494e95365c1ed4e8bb01; no voters:
I20260620 09:26:25.724797 21843 leader_election.cc:288] T 00000000000000000000000000000000 P 0b1e0d9cb04f494e95365c1ed4e8bb01 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:26:25.725020 21848 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0b1e0d9cb04f494e95365c1ed4e8bb01 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:26:25.725063 21843 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0b1e0d9cb04f494e95365c1ed4e8bb01 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:26:25.725174 21848 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0b1e0d9cb04f494e95365c1ed4e8bb01 [term 1 LEADER]: Becoming Leader. State: Replica: 0b1e0d9cb04f494e95365c1ed4e8bb01, State: Running, Role: LEADER
I20260620 09:26:25.725281 21848 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0b1e0d9cb04f494e95365c1ed4e8bb01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b1e0d9cb04f494e95365c1ed4e8bb01" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 41299 } }
I20260620 09:26:25.725633 21849 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b1e0d9cb04f494e95365c1ed4e8bb01 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0b1e0d9cb04f494e95365c1ed4e8bb01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b1e0d9cb04f494e95365c1ed4e8bb01" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 41299 } } }
I20260620 09:26:25.725625 21848 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b1e0d9cb04f494e95365c1ed4e8bb01 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0b1e0d9cb04f494e95365c1ed4e8bb01. Latest consensus state: current_term: 1 leader_uuid: "0b1e0d9cb04f494e95365c1ed4e8bb01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b1e0d9cb04f494e95365c1ed4e8bb01" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 41299 } } }
I20260620 09:26:25.725827 21848 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b1e0d9cb04f494e95365c1ed4e8bb01 [sys.catalog]: This master's current role is: LEADER
I20260620 09:26:25.725795 21849 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b1e0d9cb04f494e95365c1ed4e8bb01 [sys.catalog]: This master's current role is: LEADER
I20260620 09:26:25.726132 21857 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:26:25.726663 21857 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:26:25.728490 21857 catalog_manager.cc:1347] Generated new cluster ID: 4a014e255f1a4bb28bba79d64d4b7ae8
I20260620 09:26:25.728549 21857 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:26:25.732673 21847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:25.723507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52430 (local address 127.16.43.190:41299)
0620 09:26:25.723748 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:25.723757 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:25.723793 (+ 36us) server_negotiation.cc:407] Connection header received
0620 09:26:25.723909 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:25.723917 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:25.723989 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:25.724137 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:25.724347 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:25.725807 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:25.726994 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:25.727636 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:25.728331 (+ 695us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:25.731725 (+ 3394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:25.731728 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:25.731729 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:26:25.731946 (+ 217us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:25.731986 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:26:25.732001 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:25.732087 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:25.732216 (+ 129us) server_negotiation.cc:299] Negotiation successful
0620 09:26:25.732303 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":101,"threads_started":1}
I20260620 09:26:25.733744 16558 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 09:26:25.742570 21857 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:26:25.743248 21857 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:26:25.758204 21857 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0b1e0d9cb04f494e95365c1ed4e8bb01: Generated new TSK 0
I20260620 09:26:25.758459 21857 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 09:26:25.788404 21873 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260620 09:26:25.788511 21873 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 09:26:25.788878 21847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:25.766698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52446 (local address 127.16.43.190:41299)
0620 09:26:25.766747 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:25.766749 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:25.766847 (+ 98us) server_negotiation.cc:407] Connection header received
0620 09:26:25.766935 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:25.766936 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:25.766946 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:25.767010 (+ 64us) server_negotiation.cc:226] Negotiated authn=JWT
0620 09:26:25.767157 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:25.767743 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:25.768886 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:25.769085 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:25.769138 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:25.769329 (+ 191us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 09:26:25.788615 (+ 19286us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46143/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 09:26:25.788800 (+ 185us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52446: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46143/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":27}
W20260620 09:26:25.788954 21847 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52446: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46143/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260620 09:26:25.788973 21872 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:26:25.766604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:41299 (local address 127.0.0.1:52446)
0620 09:26:25.766782 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:25.766794 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:25.766865 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:25.767027 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:25.767028 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:25.767030 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0620 09:26:25.767095 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:25.767097 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:25.767834 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:25.767835 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:25.768799 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:25.768802 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:25.769168 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:25.769169 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:25.769231 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:25.769237 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 09:26:25.788752 (+ 19515us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46143/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 09:26:25.788863 (+ 111us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.16.43.190:41299: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46143/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":102,"threads_started":1}
I20260620 09:26:25.789301 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 21776
2026-06-20T09:26:25Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (238 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260620 09:26:25.827466 16558 mini_oidc.cc:152] Starting JWKS server
I20260620 09:26:25.828265 16558 webserver.cc:460] Webserver started at https://127.0.0.1:42719/ using document root <none> and password file <none>
I20260620 09:26:25.828337 16558 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 09:26:25.828480 16558 webserver.cc:460] Webserver started at http://127.0.0.1:37063/ using document root <none> and password file <none>
2026-06-20T09:26:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:26:25Z Disabled control of system clock
I20260620 09:26:25.836410 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781947562949448-16558-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781947562949448-16558-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781947562949448-16558-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.43.190:37003
--webserver_interface=127.16.43.190
--webserver_port=0
--builtin_ntp_servers=127.16.43.148:39359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.43.190:37003
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42719/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 09:26:25.919636 21879 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:26:25.919860 21879 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:26:25.919883 21879 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:26:25.921173 21879 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:26:25.921221 21879 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:26:25.921233 21879 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:26:25.921247 21879 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:26:25.921258 21879 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 09:26:25.921270 21879 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 09:26:25.921283 21879 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42719/jwks.json
I20260620 09:26:25.922652 21879 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:39359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781947562949448-16558-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781947562949448-16558-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.43.190:37003
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.43.190:37003
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42719/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.43.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21879
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781947562949448-16558-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:26:25.922860 21879 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:26:25.923148 21879 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:26:25.928372 21885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:26:25.928414 21879 server_base.cc:1034] running on GCE node
W20260620 09:26:25.928645 21887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:26:25.928375 21884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:26:25.929011 21879 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:26:25.929347 21879 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:26:25.930513 21879 hybrid_clock.cc:648] HybridClock initialized: now 1781947585930501 us; error 49 us; skew 500 ppm
I20260620 09:26:25.931973 21879 webserver.cc:460] Webserver started at http://127.16.43.190:46689/ using document root <none> and password file <none>
I20260620 09:26:25.932266 21879 fs_manager.cc:359] Metadata directory not provided
I20260620 09:26:25.932343 21879 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:26:25.932502 21879 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:26:25.933645 21879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781947562949448-16558-0/minicluster-data/master-0/data/instance:
uuid: "a182b350225a42dda6676e01bca2a400"
format_stamp: "Formatted at 2026-06-20 09:26:25 on dist-test-slave-706q"
server_key: "a1296a830a96f728ccae6142a2a3d946"
server_key_iv: "d44e2bc2d9fa3c87f1dd1c003eaff2d9"
server_key_version: "encryptionkey@0"
I20260620 09:26:25.934029 21879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781947562949448-16558-0/minicluster-data/master-0/wal/instance:
uuid: "a182b350225a42dda6676e01bca2a400"
format_stamp: "Formatted at 2026-06-20 09:26:25 on dist-test-slave-706q"
server_key: "a1296a830a96f728ccae6142a2a3d946"
server_key_iv: "d44e2bc2d9fa3c87f1dd1c003eaff2d9"
server_key_version: "encryptionkey@0"
I20260620 09:26:25.935437 21879 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 09:26:25.936403 21893 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:25.936802 21879 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:26:25.936941 21879 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781947562949448-16558-0/minicluster-data/master-0/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781947562949448-16558-0/minicluster-data/master-0/wal
uuid: "a182b350225a42dda6676e01bca2a400"
format_stamp: "Formatted at 2026-06-20 09:26:25 on dist-test-slave-706q"
server_key: "a1296a830a96f728ccae6142a2a3d946"
server_key_iv: "d44e2bc2d9fa3c87f1dd1c003eaff2d9"
server_key_version: "encryptionkey@0"
I20260620 09:26:25.937036 21879 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781947562949448-16558-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781947562949448-16558-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781947562949448-16558-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:26:25.969761 21879 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:26:25.970134 21879 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:26:25.970296 21879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:26:25.974352 21879 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.190:37003
I20260620 09:26:25.974388 21945 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.190:37003 every 8 connection(s)
I20260620 09:26:25.974741 21879 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
I20260620 09:26:25.975450 21946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:26:25.978237 21946 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a182b350225a42dda6676e01bca2a400: Bootstrap starting.
I20260620 09:26:25.978763 21946 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a182b350225a42dda6676e01bca2a400: Neither blocks nor log segments found. Creating new log.
I20260620 09:26:25.979004 21946 log.cc:824] T 00000000000000000000000000000000 P a182b350225a42dda6676e01bca2a400: Log is configured to *not* fsync() on all Append() calls
I20260620 09:26:25.979728 21946 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a182b350225a42dda6676e01bca2a400: No bootstrap required, opened a new log
I20260620 09:26:25.981082 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 21879
I20260620 09:26:25.981220 16558 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781947562949448-16558-0/minicluster-data/master-0/wal/instance
I20260620 09:26:25.981406 16558 external_mini_cluster.cc:1428] Setting key 8b0340a920bcdd02e6844b688889f36c
I20260620 09:26:25.981356 21946 raft_consensus.cc:348] T 00000000000000000000000000000000 P a182b350225a42dda6676e01bca2a400 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a182b350225a42dda6676e01bca2a400" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 37003 } }
I20260620 09:26:25.981516 21946 raft_consensus.cc:374] T 00000000000000000000000000000000 P a182b350225a42dda6676e01bca2a400 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:26:25.981560 21946 raft_consensus.cc:729] T 00000000000000000000000000000000 P a182b350225a42dda6676e01bca2a400 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a182b350225a42dda6676e01bca2a400, State: Initialized, Role: FOLLOWER
I20260620 09:26:25.981690 21946 consensus_queue.cc:260] T 00000000000000000000000000000000 P a182b350225a42dda6676e01bca2a400 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a182b350225a42dda6676e01bca2a400" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 37003 } }
I20260620 09:26:25.981767 21946 raft_consensus.cc:388] T 00000000000000000000000000000000 P a182b350225a42dda6676e01bca2a400 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:26:25.981798 21946 raft_consensus.cc:482] T 00000000000000000000000000000000 P a182b350225a42dda6676e01bca2a400 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:26:25.981846 21946 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a182b350225a42dda6676e01bca2a400 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:26:25.982604 21946 raft_consensus.cc:504] T 00000000000000000000000000000000 P a182b350225a42dda6676e01bca2a400 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a182b350225a42dda6676e01bca2a400" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 37003 } }
I20260620 09:26:25.982718 21946 leader_election.cc:302] T 00000000000000000000000000000000 P a182b350225a42dda6676e01bca2a400 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a182b350225a42dda6676e01bca2a400; no voters:
I20260620 09:26:25.982867 21946 leader_election.cc:288] T 00000000000000000000000000000000 P a182b350225a42dda6676e01bca2a400 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:26:25.982905 21951 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a182b350225a42dda6676e01bca2a400 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:26:25.983029 21951 raft_consensus.cc:686] T 00000000000000000000000000000000 P a182b350225a42dda6676e01bca2a400 [term 1 LEADER]: Becoming Leader. State: Replica: a182b350225a42dda6676e01bca2a400, State: Running, Role: LEADER
I20260620 09:26:25.983109 21951 consensus_queue.cc:237] T 00000000000000000000000000000000 P a182b350225a42dda6676e01bca2a400 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a182b350225a42dda6676e01bca2a400" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 37003 } }
I20260620 09:26:25.983140 21946 sys_catalog.cc:564] T 00000000000000000000000000000000 P a182b350225a42dda6676e01bca2a400 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:26:25.983718 21953 sys_catalog.cc:455] T 00000000000000000000000000000000 P a182b350225a42dda6676e01bca2a400 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a182b350225a42dda6676e01bca2a400" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a182b350225a42dda6676e01bca2a400" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 37003 } } }
I20260620 09:26:25.983798 21953 sys_catalog.cc:458] T 00000000000000000000000000000000 P a182b350225a42dda6676e01bca2a400 [sys.catalog]: This master's current role is: LEADER
I20260620 09:26:25.983960 21952 sys_catalog.cc:455] T 00000000000000000000000000000000 P a182b350225a42dda6676e01bca2a400 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a182b350225a42dda6676e01bca2a400. Latest consensus state: current_term: 1 leader_uuid: "a182b350225a42dda6676e01bca2a400" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a182b350225a42dda6676e01bca2a400" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 37003 } } }
I20260620 09:26:25.984019 21952 sys_catalog.cc:458] T 00000000000000000000000000000000 P a182b350225a42dda6676e01bca2a400 [sys.catalog]: This master's current role is: LEADER
I20260620 09:26:25.984475 21958 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:26:25.984974 21958 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:26:25.986639 21958 catalog_manager.cc:1347] Generated new cluster ID: f4c0e4a47d8442dd8b13846e9201cd53
I20260620 09:26:25.986691 21958 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:26:25.988240 21950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:25.981915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47444 (local address 127.16.43.190:37003)
0620 09:26:25.982251 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:25.982261 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:25.982293 (+ 32us) server_negotiation.cc:407] Connection header received
0620 09:26:25.982421 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:25.982428 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:25.982515 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:25.982688 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:25.982988 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:25.984971 (+ 1983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:25.985969 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:25.986574 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:25.986691 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:25.987584 (+ 893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:25.987588 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:25.987588 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:26:25.987770 (+ 182us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:25.987803 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:26:25.987814 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:25.987894 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:25.987969 (+ 75us) server_negotiation.cc:299] Negotiation successful
0620 09:26:25.988045 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":79,"threads_started":1}
I20260620 09:26:25.989076 16558 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 09:26:25.995972 21958 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:26:25.996480 21958 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:26:26.003962 21958 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a182b350225a42dda6676e01bca2a400: Generated new TSK 0
I20260620 09:26:26.004171 21958 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 09:26:26.015060 21950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:26.011195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47448 (local address 127.16.43.190:37003)
0620 09:26:26.011242 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:26.011244 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:26.011329 (+ 85us) server_negotiation.cc:407] Connection header received
0620 09:26:26.011412 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:26.011413 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:26.011433 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:26.011494 (+ 61us) server_negotiation.cc:226] Negotiated authn=JWT
0620 09:26:26.011677 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:26.012177 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:26.013359 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:26.013556 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:26.013607 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:26.013810 (+ 203us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 09:26:26.014831 (+ 1021us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42719/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 09:26:26.014978 (+ 147us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:47448: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42719/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":23}
W20260620 09:26:26.015128 21950 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:47448: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42719/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260620 09:26:26.017197 21975 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:26:26.011084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:37003 (local address 127.0.0.1:47448)
0620 09:26:26.011261 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:26.011273 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:26.011342 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:26.011513 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:26.011514 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:26.011516 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0620 09:26:26.011612 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:26.011615 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:26.012260 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:26.012261 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:26.013263 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:26.013267 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:26.013640 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:26.013641 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:26.013705 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:26.013712 (+ 7us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 09:26:26.014970 (+ 1258us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42719/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 09:26:26.017082 (+ 2112us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.16.43.190:37003: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42719/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":147,"mutex_wait_us":1852,"thread_start_us":93,"threads_started":1}
I20260620 09:26:26.017405 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 21879
E20260620 09:26:26.026046 21976 webserver.cc:533] Webserver: sslize failed:
E20260620 09:26:26.026145 21976 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-06-20T09:26:26Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (227 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-20T09:26:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:26:26Z Disabled control of system clock
I20260620 09:26:26.063462 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781947562949448-16558-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781947562949448-16558-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781947562949448-16558-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.43.190:42237
--webserver_interface=127.16.43.190
--webserver_port=0
--builtin_ntp_servers=127.16.43.148:34095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.43.190:42237
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781947562949448-16558-0/testchainca.cert with env {}
W20260620 09:26:26.144380 21980 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:26:26.144639 21980 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:26:26.144677 21980 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:26:26.145917 21980 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:26:26.145990 21980 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:26:26.146010 21980 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:26:26.146029 21980 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:26:26.146056 21980 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 09:26:26.146086 21980 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 09:26:26.146106 21980 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260620 09:26:26.147694 21980 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:34095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781947562949448-16558-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781947562949448-16558-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.43.190:42237
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.43.190:42237
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.43.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781947562949448-16558-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21980
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781947562949448-16558-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:26:26.147972 21980 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:26:26.148190 21980 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:26:26.153357 21986 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:26:26.153378 21988 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:26:26.153394 21980 server_base.cc:1034] running on GCE node
W20260620 09:26:26.153389 21985 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:26:26.153826 21980 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:26:26.154067 21980 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:26:26.155231 21980 hybrid_clock.cc:648] HybridClock initialized: now 1781947586155208 us; error 34 us; skew 500 ppm
I20260620 09:26:26.156524 21980 webserver.cc:460] Webserver started at http://127.16.43.190:36737/ using document root <none> and password file <none>
I20260620 09:26:26.156769 21980 fs_manager.cc:359] Metadata directory not provided
I20260620 09:26:26.156842 21980 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:26:26.156977 21980 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:26:26.158114 21980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781947562949448-16558-0/minicluster-data/master-0/data/instance:
uuid: "f7d856e9c54c4b5992ec1d0bd33f8960"
format_stamp: "Formatted at 2026-06-20 09:26:26 on dist-test-slave-706q"
server_key: "50ceb0e54d522ca7043274fbfe60939c"
server_key_iv: "aecedc005945e340db6c4debae65e30b"
server_key_version: "encryptionkey@0"
I20260620 09:26:26.158489 21980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781947562949448-16558-0/minicluster-data/master-0/wal/instance:
uuid: "f7d856e9c54c4b5992ec1d0bd33f8960"
format_stamp: "Formatted at 2026-06-20 09:26:26 on dist-test-slave-706q"
server_key: "50ceb0e54d522ca7043274fbfe60939c"
server_key_iv: "aecedc005945e340db6c4debae65e30b"
server_key_version: "encryptionkey@0"
I20260620 09:26:26.159855 21980 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 09:26:26.160557 21994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:26.160717 21980 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 09:26:26.160821 21980 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781947562949448-16558-0/minicluster-data/master-0/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781947562949448-16558-0/minicluster-data/master-0/wal
uuid: "f7d856e9c54c4b5992ec1d0bd33f8960"
format_stamp: "Formatted at 2026-06-20 09:26:26 on dist-test-slave-706q"
server_key: "50ceb0e54d522ca7043274fbfe60939c"
server_key_iv: "aecedc005945e340db6c4debae65e30b"
server_key_version: "encryptionkey@0"
I20260620 09:26:26.160894 21980 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781947562949448-16558-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781947562949448-16558-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781947562949448-16558-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:26:26.186195 21980 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:26:26.186547 21980 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:26:26.186709 21980 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:26:26.190603 21980 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.190:42237
I20260620 09:26:26.190670 22046 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.190:42237 every 8 connection(s)
I20260620 09:26:26.191025 21980 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
I20260620 09:26:26.192153 22047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:26:26.194535 22047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7d856e9c54c4b5992ec1d0bd33f8960: Bootstrap starting.
I20260620 09:26:26.195145 22047 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f7d856e9c54c4b5992ec1d0bd33f8960: Neither blocks nor log segments found. Creating new log.
I20260620 09:26:26.195429 22047 log.cc:824] T 00000000000000000000000000000000 P f7d856e9c54c4b5992ec1d0bd33f8960: Log is configured to *not* fsync() on all Append() calls
I20260620 09:26:26.196161 22047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7d856e9c54c4b5992ec1d0bd33f8960: No bootstrap required, opened a new log
I20260620 09:26:26.197826 22047 raft_consensus.cc:348] T 00000000000000000000000000000000 P f7d856e9c54c4b5992ec1d0bd33f8960 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7d856e9c54c4b5992ec1d0bd33f8960" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 42237 } }
I20260620 09:26:26.197969 22047 raft_consensus.cc:374] T 00000000000000000000000000000000 P f7d856e9c54c4b5992ec1d0bd33f8960 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:26:26.197992 22047 raft_consensus.cc:729] T 00000000000000000000000000000000 P f7d856e9c54c4b5992ec1d0bd33f8960 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7d856e9c54c4b5992ec1d0bd33f8960, State: Initialized, Role: FOLLOWER
I20260620 09:26:26.198011 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 21980
I20260620 09:26:26.198145 16558 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781947562949448-16558-0/minicluster-data/master-0/wal/instance
I20260620 09:26:26.198144 22047 consensus_queue.cc:260] T 00000000000000000000000000000000 P f7d856e9c54c4b5992ec1d0bd33f8960 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7d856e9c54c4b5992ec1d0bd33f8960" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 42237 } }
I20260620 09:26:26.198257 22047 raft_consensus.cc:388] T 00000000000000000000000000000000 P f7d856e9c54c4b5992ec1d0bd33f8960 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:26:26.198297 22047 raft_consensus.cc:482] T 00000000000000000000000000000000 P f7d856e9c54c4b5992ec1d0bd33f8960 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:26:26.198343 16558 external_mini_cluster.cc:1428] Setting key 7ae49acf6778068d2e185ed1d44ab9b6
I20260620 09:26:26.198364 22047 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f7d856e9c54c4b5992ec1d0bd33f8960 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:26:26.199147 22047 raft_consensus.cc:504] T 00000000000000000000000000000000 P f7d856e9c54c4b5992ec1d0bd33f8960 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7d856e9c54c4b5992ec1d0bd33f8960" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 42237 } }
I20260620 09:26:26.199256 22047 leader_election.cc:302] T 00000000000000000000000000000000 P f7d856e9c54c4b5992ec1d0bd33f8960 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f7d856e9c54c4b5992ec1d0bd33f8960; no voters:
I20260620 09:26:26.199411 22047 leader_election.cc:288] T 00000000000000000000000000000000 P f7d856e9c54c4b5992ec1d0bd33f8960 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:26:26.199467 22052 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f7d856e9c54c4b5992ec1d0bd33f8960 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:26:26.199625 22052 raft_consensus.cc:686] T 00000000000000000000000000000000 P f7d856e9c54c4b5992ec1d0bd33f8960 [term 1 LEADER]: Becoming Leader. State: Replica: f7d856e9c54c4b5992ec1d0bd33f8960, State: Running, Role: LEADER
I20260620 09:26:26.199702 22047 sys_catalog.cc:564] T 00000000000000000000000000000000 P f7d856e9c54c4b5992ec1d0bd33f8960 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:26:26.199734 22052 consensus_queue.cc:237] T 00000000000000000000000000000000 P f7d856e9c54c4b5992ec1d0bd33f8960 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7d856e9c54c4b5992ec1d0bd33f8960" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 42237 } }
I20260620 09:26:26.200068 22052 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7d856e9c54c4b5992ec1d0bd33f8960 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f7d856e9c54c4b5992ec1d0bd33f8960. Latest consensus state: current_term: 1 leader_uuid: "f7d856e9c54c4b5992ec1d0bd33f8960" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7d856e9c54c4b5992ec1d0bd33f8960" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 42237 } } }
I20260620 09:26:26.200141 22052 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7d856e9c54c4b5992ec1d0bd33f8960 [sys.catalog]: This master's current role is: LEADER
I20260620 09:26:26.200285 22054 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7d856e9c54c4b5992ec1d0bd33f8960 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f7d856e9c54c4b5992ec1d0bd33f8960" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7d856e9c54c4b5992ec1d0bd33f8960" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 42237 } } }
I20260620 09:26:26.200352 22054 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7d856e9c54c4b5992ec1d0bd33f8960 [sys.catalog]: This master's current role is: LEADER
I20260620 09:26:26.200465 22056 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:26:26.201186 22056 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:26:26.202836 22056 catalog_manager.cc:1347] Generated new cluster ID: 5719596505224861852e55956597d8e6
I20260620 09:26:26.202904 22056 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:26:26.204991 22051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:26.198875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58218 (local address 127.16.43.190:42237)
0620 09:26:26.199118 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:26.199128 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:26.199166 (+ 38us) server_negotiation.cc:407] Connection header received
0620 09:26:26.199286 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:26.199293 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:26.199371 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:26.199546 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:26.199915 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:26.201755 (+ 1840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:26.202787 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:26.203381 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:26.203505 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:26.204273 (+ 768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:26.204277 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:26.204277 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:26:26.204496 (+ 219us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:26.204543 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:26:26.204558 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:26.204633 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:26.204701 (+ 68us) server_negotiation.cc:299] Negotiation successful
0620 09:26:26.204783 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":94,"threads_started":1}
I20260620 09:26:26.206007 16558 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 09:26:26.208400 22056 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:26:26.209079 22056 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:26:26.220834 22056 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f7d856e9c54c4b5992ec1d0bd33f8960: Generated new TSK 0
I20260620 09:26:26.221050 22056 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 09:26:26.269016 22051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:26.230600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58222 (local address 127.16.43.190:42237)
0620 09:26:26.230659 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:26.230662 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:26.230753 (+ 91us) server_negotiation.cc:407] Connection header received
0620 09:26:26.230812 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:26.230813 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:26.230826 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:26.230909 (+ 83us) server_negotiation.cc:226] Negotiated authn=JWT
0620 09:26:26.231052 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:26.231911 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:26.232682 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:26.233019 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:26.233091 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:26.233260 (+ 169us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 09:26:26.268737 (+ 35477us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 09:26:26.268935 (+ 198us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58222: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":36}
W20260620 09:26:26.269112 22051 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58222: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260620 09:26:26.269145 22076 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:26:26.230505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:42237 (local address 127.0.0.1:58222)
0620 09:26:26.230683 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:26.230701 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:26.230756 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:26.230920 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:26.230920 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:26.230923 (+ 3us) client_negotiation.cc:189] Negotiated authn=JWT
0620 09:26:26.230982 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:26.230985 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:26.232011 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:26.232012 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:26.232601 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:26.232603 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:26.233100 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:26.233101 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:26.233157 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:26.233163 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 09:26:26.268891 (+ 35728us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 09:26:26.269036 (+ 145us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.16.43.190:42237: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":113,"threads_started":1}
I20260620 09:26:26.269580 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 21980
2026-06-20T09:26:26Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (231 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (92 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (103 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781947562949448-16558-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:26:26 dist-test-slave-706q krb5kdc[22088](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:26:26 dist-test-slave-706q krb5kdc[22088](info): set up 2 sockets
Jun 20 09:26:26 dist-test-slave-706q krb5kdc[22088](info): commencing operation
krb5kdc: starting...
W20260620 09:26:27.504766 16558 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781947562949448-16558-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781947562949448-16558-0/krb5kdc/test-user.keytab.
Jun 20 09:26:27 dist-test-slave-706q krb5kdc[22088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947587, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:26:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:26:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.43.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.43.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.43.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.190.keytab.
Entry for principal kudu/127.16.43.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.190.keytab.
I20260620 09:26:27.622520 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781947562949448-16558-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781947562949448-16558-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781947562949448-16558-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.43.190:37107
--webserver_interface=127.16.43.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.190.keytab
--principal=kudu/127.16.43.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.43.148:35725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.43.190:37107
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781947562949448-16558-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781947562949448-16558-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781947562949448-16558-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:26:27.703411 22101 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:26:27.703650 22101 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:26:27.703673 22101 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:26:27.704882 22101 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:26:27.704926 22101 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:26:27.704941 22101 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:26:27.704953 22101 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:26:27.704965 22101 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:26:27.706421 22101 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:35725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781947562949448-16558-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781947562949448-16558-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.43.190:37107
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.190.keytab
--principal=kudu/127.16.43.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.43.190:37107
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.43.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22101
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781947562949448-16558-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:26:27.706666 22101 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:26:27.706893 22101 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:26:27.711877 22107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:26:27.711871 22109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:26:27.711905 22106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:26:27.711927 22101 server_base.cc:1034] running on GCE node
I20260620 09:26:27.712342 22101 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:26:27.712675 22101 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:26:27.713864 22101 hybrid_clock.cc:648] HybridClock initialized: now 1781947587713859 us; error 42 us; skew 500 ppm
Jun 20 09:26:27 dist-test-slave-706q krb5kdc[22088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947587, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:26:27.719496 22101 init.cc:377] Logged in from keytab as kudu/127.16.43.190@KRBTEST.COM (short username kudu)
I20260620 09:26:27.720070 22101 webserver.cc:460] Webserver started at http://127.16.43.190:45533/ using document root <none> and password file <none>
I20260620 09:26:27.720338 22101 fs_manager.cc:359] Metadata directory not provided
I20260620 09:26:27.720408 22101 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:26:27.720562 22101 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:26:27.721661 22101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781947562949448-16558-0/minicluster-data/master-0/data/instance:
uuid: "75e00e56f6274a7494f6bf1d63633adb"
format_stamp: "Formatted at 2026-06-20 09:26:27 on dist-test-slave-706q"
server_key: "00576b015c439d1b4e20779fa54d673a"
server_key_iv: "c0fa5dc6918bdb7073b4b429157cdfba"
server_key_version: "encryptionkey@0"
I20260620 09:26:27.722049 22101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781947562949448-16558-0/minicluster-data/master-0/wal/instance:
uuid: "75e00e56f6274a7494f6bf1d63633adb"
format_stamp: "Formatted at 2026-06-20 09:26:27 on dist-test-slave-706q"
server_key: "00576b015c439d1b4e20779fa54d673a"
server_key_iv: "c0fa5dc6918bdb7073b4b429157cdfba"
server_key_version: "encryptionkey@0"
I20260620 09:26:27.723420 22101 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 09:26:27.724393 22116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:27.724630 22101 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 09:26:27.724749 22101 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781947562949448-16558-0/minicluster-data/master-0/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781947562949448-16558-0/minicluster-data/master-0/wal
uuid: "75e00e56f6274a7494f6bf1d63633adb"
format_stamp: "Formatted at 2026-06-20 09:26:27 on dist-test-slave-706q"
server_key: "00576b015c439d1b4e20779fa54d673a"
server_key_iv: "c0fa5dc6918bdb7073b4b429157cdfba"
server_key_version: "encryptionkey@0"
I20260620 09:26:27.724825 22101 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781947562949448-16558-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781947562949448-16558-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781947562949448-16558-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:26:27.738082 22101 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:26:27.739992 22101 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:26:27.740163 22101 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:26:27.744351 22101 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.190:37107
I20260620 09:26:27.744390 22168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.190:37107 every 8 connection(s)
I20260620 09:26:27.744761 22101 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
I20260620 09:26:27.745664 22169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:26:27.747516 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 22101
I20260620 09:26:27.747674 16558 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781947562949448-16558-0/minicluster-data/master-0/wal/instance
I20260620 09:26:27.747864 16558 external_mini_cluster.cc:1428] Setting key 2a7d412b7669b731640a5db58f674d10
I20260620 09:26:27.748384 22169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75e00e56f6274a7494f6bf1d63633adb: Bootstrap starting.
I20260620 09:26:27.749094 22169 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 75e00e56f6274a7494f6bf1d63633adb: Neither blocks nor log segments found. Creating new log.
I20260620 09:26:27.749421 22169 log.cc:824] T 00000000000000000000000000000000 P 75e00e56f6274a7494f6bf1d63633adb: Log is configured to *not* fsync() on all Append() calls
I20260620 09:26:27.750144 22169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75e00e56f6274a7494f6bf1d63633adb: No bootstrap required, opened a new log
I20260620 09:26:27.752102 22169 raft_consensus.cc:348] T 00000000000000000000000000000000 P 75e00e56f6274a7494f6bf1d63633adb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75e00e56f6274a7494f6bf1d63633adb" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 37107 } }
I20260620 09:26:27.752269 22169 raft_consensus.cc:374] T 00000000000000000000000000000000 P 75e00e56f6274a7494f6bf1d63633adb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:26:27.752316 22169 raft_consensus.cc:729] T 00000000000000000000000000000000 P 75e00e56f6274a7494f6bf1d63633adb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75e00e56f6274a7494f6bf1d63633adb, State: Initialized, Role: FOLLOWER
I20260620 09:26:27.752453 22169 consensus_queue.cc:260] T 00000000000000000000000000000000 P 75e00e56f6274a7494f6bf1d63633adb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75e00e56f6274a7494f6bf1d63633adb" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 37107 } }
I20260620 09:26:27.752547 22169 raft_consensus.cc:388] T 00000000000000000000000000000000 P 75e00e56f6274a7494f6bf1d63633adb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:26:27.752591 22169 raft_consensus.cc:482] T 00000000000000000000000000000000 P 75e00e56f6274a7494f6bf1d63633adb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:26:27.752648 22169 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 75e00e56f6274a7494f6bf1d63633adb [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:26:27.753298 22169 raft_consensus.cc:504] T 00000000000000000000000000000000 P 75e00e56f6274a7494f6bf1d63633adb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75e00e56f6274a7494f6bf1d63633adb" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 37107 } }
I20260620 09:26:27.753414 22169 leader_election.cc:302] T 00000000000000000000000000000000 P 75e00e56f6274a7494f6bf1d63633adb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 75e00e56f6274a7494f6bf1d63633adb; no voters:
I20260620 09:26:27.753605 22169 leader_election.cc:288] T 00000000000000000000000000000000 P 75e00e56f6274a7494f6bf1d63633adb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:26:27.753664 22174 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 75e00e56f6274a7494f6bf1d63633adb [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:26:27.753840 22174 raft_consensus.cc:686] T 00000000000000000000000000000000 P 75e00e56f6274a7494f6bf1d63633adb [term 1 LEADER]: Becoming Leader. State: Replica: 75e00e56f6274a7494f6bf1d63633adb, State: Running, Role: LEADER
I20260620 09:26:27.753903 22169 sys_catalog.cc:564] T 00000000000000000000000000000000 P 75e00e56f6274a7494f6bf1d63633adb [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:26:27.753935 22174 consensus_queue.cc:237] T 00000000000000000000000000000000 P 75e00e56f6274a7494f6bf1d63633adb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75e00e56f6274a7494f6bf1d63633adb" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 37107 } }
I20260620 09:26:27.754284 22174 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75e00e56f6274a7494f6bf1d63633adb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 75e00e56f6274a7494f6bf1d63633adb. Latest consensus state: current_term: 1 leader_uuid: "75e00e56f6274a7494f6bf1d63633adb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75e00e56f6274a7494f6bf1d63633adb" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 37107 } } }
I20260620 09:26:27.754355 22174 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75e00e56f6274a7494f6bf1d63633adb [sys.catalog]: This master's current role is: LEADER
I20260620 09:26:27.754299 22176 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75e00e56f6274a7494f6bf1d63633adb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "75e00e56f6274a7494f6bf1d63633adb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75e00e56f6274a7494f6bf1d63633adb" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 37107 } } }
I20260620 09:26:27.754521 22176 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75e00e56f6274a7494f6bf1d63633adb [sys.catalog]: This master's current role is: LEADER
I20260620 09:26:27.754576 22181 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:26:27.755090 22181 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 20 09:26:27 dist-test-slave-706q krb5kdc[22088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947587, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.43.190@KRBTEST.COM
I20260620 09:26:27.759368 22172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:27.748418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38090 (local address 127.16.43.190:37107)
0620 09:26:27.748595 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:27.748605 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:27.748638 (+ 33us) server_negotiation.cc:407] Connection header received
0620 09:26:27.749289 (+ 651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:27.749298 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:27.749367 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:27.749527 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:27.750348 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:27.751278 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:27.752133 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:27.752693 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:27.752814 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:27.755826 (+ 3012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:27.755830 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:27.755830 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:26:27.755843 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:27.757765 (+ 1922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:27.758111 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:27.758115 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:27.758115 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:27.758230 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:27.758423 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:27.758424 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:27.758424 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:27.758728 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:27.758868 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:27.758988 (+ 120us) server_negotiation.cc:299] Negotiation successful
0620 09:26:27.759126 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":87,"threads_started":1}
I20260620 09:26:27.760604 22181 catalog_manager.cc:1347] Generated new cluster ID: 138c2e1b65fd4064b79b4bab345b4a05
I20260620 09:26:27.760689 22181 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:26:27.774591 22181 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:26:27.775164 22181 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:26:27.781422 22181 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 75e00e56f6274a7494f6bf1d63633adb: Generated new TSK 0
I20260620 09:26:27.781633 22181 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.43.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.43.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.43.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.129.keytab.
Entry for principal kudu/127.16.43.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.129.keytab.
I20260620 09:26:27.810182 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781947562949448-16558-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781947562949448-16558-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781947562949448-16558-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781947562949448-16558-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.43.129:0
--local_ip_for_outbound_sockets=127.16.43.129
--webserver_interface=127.16.43.129
--webserver_port=0
--tserver_master_addrs=127.16.43.190:37107
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.129.keytab
--principal=kudu/127.16.43.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.43.148:35725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781947562949448-16558-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781947562949448-16558-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781947562949448-16558-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:26:27.893673 22195 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:26:27.893942 22195 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:26:27.893983 22195 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:26:27.895186 22195 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:26:27.895260 22195 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:26:27.895314 22195 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.43.129
I20260620 09:26:27.896862 22195 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:35725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781947562949448-16558-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781947562949448-16558-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.129.keytab
--principal=kudu/127.16.43.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.43.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781947562949448-16558-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.43.129
--webserver_port=0
--tserver_master_addrs=127.16.43.190:37107
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22195
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.43.129
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781947562949448-16558-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:26:27.897140 22195 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:26:27.897395 22195 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:26:27.902688 22200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:26:27.902684 22203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:26:27.902704 22201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:26:27.903038 22195 server_base.cc:1034] running on GCE node
I20260620 09:26:27.903275 22195 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:26:27.903479 22195 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:26:27.904650 22195 hybrid_clock.cc:648] HybridClock initialized: now 1781947587904637 us; error 32 us; skew 500 ppm
Jun 20 09:26:27 dist-test-slave-706q krb5kdc[22088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947587, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:26:27.907245 22195 init.cc:377] Logged in from keytab as kudu/127.16.43.129@KRBTEST.COM (short username kudu)
I20260620 09:26:27.907750 22195 webserver.cc:460] Webserver started at http://127.16.43.129:42411/ using document root <none> and password file <none>
I20260620 09:26:27.908020 22195 fs_manager.cc:359] Metadata directory not provided
I20260620 09:26:27.908092 22195 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:26:27.908233 22195 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:26:27.909281 22195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781947562949448-16558-0/minicluster-data/ts-0/data/instance:
uuid: "04020650729648d3bc36f514a74fc4c5"
format_stamp: "Formatted at 2026-06-20 09:26:27 on dist-test-slave-706q"
server_key: "2ea75105b6909aab38d7b2d8301265b3"
server_key_iv: "3a02e09d66662792e27d9cfe33980dcc"
server_key_version: "encryptionkey@0"
I20260620 09:26:27.909583 22195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781947562949448-16558-0/minicluster-data/ts-0/wal/instance:
uuid: "04020650729648d3bc36f514a74fc4c5"
format_stamp: "Formatted at 2026-06-20 09:26:27 on dist-test-slave-706q"
server_key: "2ea75105b6909aab38d7b2d8301265b3"
server_key_iv: "3a02e09d66662792e27d9cfe33980dcc"
server_key_version: "encryptionkey@0"
I20260620 09:26:27.910934 22195 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 09:26:27.911733 22210 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:27.911938 22195 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:26:27.912015 22195 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781947562949448-16558-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781947562949448-16558-0/minicluster-data/ts-0/wal
uuid: "04020650729648d3bc36f514a74fc4c5"
format_stamp: "Formatted at 2026-06-20 09:26:27 on dist-test-slave-706q"
server_key: "2ea75105b6909aab38d7b2d8301265b3"
server_key_iv: "3a02e09d66662792e27d9cfe33980dcc"
server_key_version: "encryptionkey@0"
I20260620 09:26:27.912074 22195 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781947562949448-16558-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781947562949448-16558-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781947562949448-16558-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:26:27.932971 22195 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:26:27.935086 22195 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:26:27.935214 22195 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:26:27.935402 22195 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:26:27.935786 22195 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:26:27.935825 22195 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:27.935848 22195 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:26:27.935861 22195 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:27.941689 22195 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.129:39365
I20260620 09:26:27.941810 22323 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.129:39365 every 8 connection(s)
I20260620 09:26:27.942116 22195 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781947562949448-16558-0/minicluster-data/ts-0/data/info.pb
I20260620 09:26:27.945840 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 22195
I20260620 09:26:27.945935 16558 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781947562949448-16558-0/minicluster-data/ts-0/wal/instance
I20260620 09:26:27.946143 16558 external_mini_cluster.cc:1428] Setting key 048d7b2f9cbab08112fd98f21a384f99
Jun 20 09:26:27 dist-test-slave-706q krb5kdc[22088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947587, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.129@KRBTEST.COM for kudu/127.16.43.190@KRBTEST.COM
I20260620 09:26:27.952765 22172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:27.942941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.129:46755 (local address 127.16.43.190:37107)
0620 09:26:27.943029 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:27.943032 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:27.943205 (+ 173us) server_negotiation.cc:407] Connection header received
0620 09:26:27.944158 (+ 953us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:27.944160 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:27.944171 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:27.944235 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:27.944910 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:27.945433 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:27.946649 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:27.946886 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:27.946974 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:27.949580 (+ 2606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:27.949582 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:27.949582 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:26:27.949591 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:27.951211 (+ 1620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:27.951756 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:27.951757 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:27.951758 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:27.951823 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:27.952087 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:27.952088 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:27.952088 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:27.952221 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:27.952296 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:27.952586 (+ 290us) server_negotiation.cc:299] Negotiation successful
0620 09:26:27.952672 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 09:26:27.952970 22326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:27.942888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:37107 (local address 127.16.43.129:46755)
0620 09:26:27.943076 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:27.943115 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:27.944047 (+ 932us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:27.944312 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:27.944316 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:27.944580 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:27.944800 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:27.944804 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:27.945511 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:27.945512 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:27.946552 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:27.946555 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:27.946985 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:27.946986 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:27.947138 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:27.947712 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:26:27.947734 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:26:27.949480 (+ 1746us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:26:27.951308 (+ 1828us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:27.951315 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:27.951317 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:27.951621 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:27.951897 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:27.951898 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:27.951898 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:27.952011 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:27.952301 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:26:27.952308 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:26:27.952444 (+ 136us) client_negotiation.cc:769] Sending connection context
0620 09:26:27.952548 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 09:26:27.952721 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":109,"threads_started":1}
I20260620 09:26:27.953497 22324 heartbeater.cc:347] Connected to a master server at 127.16.43.190:37107
I20260620 09:26:27.953621 22324 heartbeater.cc:464] Registering TS with master...
I20260620 09:26:27.953833 22324 heartbeater.cc:511] Master 127.16.43.190:37107 requested a full tablet report, sending...
I20260620 09:26:27.954509 22133 ts_manager.cc:194] Registered new tserver with Master: 04020650729648d3bc36f514a74fc4c5 (127.16.43.129:39365)
WARNING: no policy specified for kudu/127.16.43.130@KRBTEST.COM; defaulting to no policy
I20260620 09:26:27.955485 22133 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.43.129@KRBTEST.COM'} at 127.16.43.129:46755
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.43.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.43.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.130.keytab.
Entry for principal kudu/127.16.43.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.130.keytab.
I20260620 09:26:27.972581 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781947562949448-16558-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781947562949448-16558-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781947562949448-16558-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781947562949448-16558-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.43.130:0
--local_ip_for_outbound_sockets=127.16.43.130
--webserver_interface=127.16.43.130
--webserver_port=0
--tserver_master_addrs=127.16.43.190:37107
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.130.keytab
--principal=kudu/127.16.43.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.43.148:35725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781947562949448-16558-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781947562949448-16558-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781947562949448-16558-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:26:28.057158 22329 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:26:28.057370 22329 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:26:28.057394 22329 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:26:28.058604 22329 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:26:28.058651 22329 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:26:28.058677 22329 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.43.130
I20260620 09:26:28.061173 22329 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:35725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781947562949448-16558-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781947562949448-16558-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.130.keytab
--principal=kudu/127.16.43.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.43.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781947562949448-16558-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.43.130
--webserver_port=0
--tserver_master_addrs=127.16.43.190:37107
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22329
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.43.130
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781947562949448-16558-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:26:28.061450 22329 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:26:28.061771 22329 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:26:28.067220 22337 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:26:28.067206 22334 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:26:28.067211 22335 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:26:28.067375 22329 server_base.cc:1034] running on GCE node
I20260620 09:26:28.067979 22329 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:26:28.068290 22329 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:26:28.069451 22329 hybrid_clock.cc:648] HybridClock initialized: now 1781947588069439 us; error 33 us; skew 500 ppm
Jun 20 09:26:28 dist-test-slave-706q krb5kdc[22088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947588, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:26:28.072214 22329 init.cc:377] Logged in from keytab as kudu/127.16.43.130@KRBTEST.COM (short username kudu)
I20260620 09:26:28.072762 22329 webserver.cc:460] Webserver started at http://127.16.43.130:39317/ using document root <none> and password file <none>
I20260620 09:26:28.073017 22329 fs_manager.cc:359] Metadata directory not provided
I20260620 09:26:28.073105 22329 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:26:28.073267 22329 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:26:28.074398 22329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781947562949448-16558-0/minicluster-data/ts-1/data/instance:
uuid: "8d1e7e6c30c04bc8ada07560c69f753e"
format_stamp: "Formatted at 2026-06-20 09:26:28 on dist-test-slave-706q"
server_key: "27fe87c390e52c7dbf41807a2d3419e2"
server_key_iv: "2b4c48ddea57f19e8d967d71c688f13f"
server_key_version: "encryptionkey@0"
I20260620 09:26:28.074805 22329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781947562949448-16558-0/minicluster-data/ts-1/wal/instance:
uuid: "8d1e7e6c30c04bc8ada07560c69f753e"
format_stamp: "Formatted at 2026-06-20 09:26:28 on dist-test-slave-706q"
server_key: "27fe87c390e52c7dbf41807a2d3419e2"
server_key_iv: "2b4c48ddea57f19e8d967d71c688f13f"
server_key_version: "encryptionkey@0"
I20260620 09:26:28.076403 22329 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 09:26:28.077291 22344 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:28.077529 22329 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:26:28.077706 22329 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781947562949448-16558-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781947562949448-16558-0/minicluster-data/ts-1/wal
uuid: "8d1e7e6c30c04bc8ada07560c69f753e"
format_stamp: "Formatted at 2026-06-20 09:26:28 on dist-test-slave-706q"
server_key: "27fe87c390e52c7dbf41807a2d3419e2"
server_key_iv: "2b4c48ddea57f19e8d967d71c688f13f"
server_key_version: "encryptionkey@0"
I20260620 09:26:28.077795 22329 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781947562949448-16558-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781947562949448-16558-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781947562949448-16558-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:26:28.099164 22329 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:26:28.101372 22329 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:26:28.101552 22329 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:26:28.101814 22329 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:26:28.102134 22329 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:26:28.102190 22329 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:28.102226 22329 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:26:28.102272 22329 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:28.108244 22329 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.130:40067
I20260620 09:26:28.108326 22457 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.130:40067 every 8 connection(s)
I20260620 09:26:28.108672 22329 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781947562949448-16558-0/minicluster-data/ts-1/data/info.pb
Jun 20 09:26:28 dist-test-slave-706q krb5kdc[22088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947588, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.130@KRBTEST.COM for kudu/127.16.43.190@KRBTEST.COM
I20260620 09:26:28.118369 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 22329
I20260620 09:26:28.118474 16558 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781947562949448-16558-0/minicluster-data/ts-1/wal/instance
I20260620 09:26:28.118686 16558 external_mini_cluster.cc:1428] Setting key 0dd4ade9bacf0657956baa50071e33c8
I20260620 09:26:28.119421 22172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:28.109528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.130:43245 (local address 127.16.43.190:37107)
0620 09:26:28.109586 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:28.109589 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:28.109906 (+ 317us) server_negotiation.cc:407] Connection header received
0620 09:26:28.111124 (+ 1218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:28.111125 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:28.111138 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:28.111216 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:28.112004 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:28.112555 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:28.113671 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:28.113876 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:28.113940 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:28.116250 (+ 2310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:28.116252 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:28.116252 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:26:28.116260 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:28.117709 (+ 1449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:28.118119 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:28.118121 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:28.118121 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:28.118184 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:28.118510 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:28.118512 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:28.118512 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:28.118710 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:28.118819 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:28.119169 (+ 350us) server_negotiation.cc:299] Negotiation successful
0620 09:26:28.119331 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 09:26:28.119587 22460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:28.109441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:37107 (local address 127.16.43.130:43245)
0620 09:26:28.109747 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:28.109797 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:28.111000 (+ 1203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:28.111290 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:28.111294 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:28.111675 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:28.111886 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:28.111891 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:28.112694 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:28.112695 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:28.113527 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:28.113529 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:28.113962 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:28.113963 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:28.114065 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:28.114620 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:26:28.114629 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:26:28.116160 (+ 1531us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:26:28.117792 (+ 1632us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:28.117798 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:28.117800 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:28.118031 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:28.118262 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:28.118262 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:28.118263 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:28.118410 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:28.118809 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:26:28.118816 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:26:28.118990 (+ 174us) client_negotiation.cc:769] Sending connection context
0620 09:26:28.119146 (+ 156us) client_negotiation.cc:240] Negotiation successful
0620 09:26:28.119326 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":103,"threads_started":1}
I20260620 09:26:28.120023 22458 heartbeater.cc:347] Connected to a master server at 127.16.43.190:37107
I20260620 09:26:28.120160 22458 heartbeater.cc:464] Registering TS with master...
I20260620 09:26:28.120358 22458 heartbeater.cc:511] Master 127.16.43.190:37107 requested a full tablet report, sending...
I20260620 09:26:28.120752 22133 ts_manager.cc:194] Registered new tserver with Master: 8d1e7e6c30c04bc8ada07560c69f753e (127.16.43.130:40067)
I20260620 09:26:28.121308 22133 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.43.130@KRBTEST.COM'} at 127.16.43.130:43245
WARNING: no policy specified for kudu/127.16.43.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.43.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.43.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.131.keytab.
Entry for principal kudu/127.16.43.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.131.keytab.
I20260620 09:26:28.147169 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781947562949448-16558-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781947562949448-16558-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781947562949448-16558-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781947562949448-16558-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.43.131:0
--local_ip_for_outbound_sockets=127.16.43.131
--webserver_interface=127.16.43.131
--webserver_port=0
--tserver_master_addrs=127.16.43.190:37107
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.131.keytab
--principal=kudu/127.16.43.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.43.148:35725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781947562949448-16558-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781947562949448-16558-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781947562949448-16558-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:26:28.229084 22463 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:26:28.229327 22463 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:26:28.229363 22463 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:26:28.230548 22463 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:26:28.230618 22463 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:26:28.230659 22463 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.43.131
I20260620 09:26:28.232206 22463 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:35725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781947562949448-16558-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781947562949448-16558-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.131.keytab
--principal=kudu/127.16.43.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.43.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781947562949448-16558-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.43.131
--webserver_port=0
--tserver_master_addrs=127.16.43.190:37107
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22463
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.43.131
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781947562949448-16558-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:26:28.232487 22463 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:26:28.232731 22463 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:26:28.237937 22469 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:26:28.237938 22471 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:26:28.238003 22468 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:26:28.238479 22463 server_base.cc:1034] running on GCE node
I20260620 09:26:28.238773 22463 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:26:28.238991 22463 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:26:28.240154 22463 hybrid_clock.cc:648] HybridClock initialized: now 1781947588240138 us; error 29 us; skew 500 ppm
Jun 20 09:26:28 dist-test-slave-706q krb5kdc[22088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947588, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:26:28.242810 22463 init.cc:377] Logged in from keytab as kudu/127.16.43.131@KRBTEST.COM (short username kudu)
I20260620 09:26:28.243330 22463 webserver.cc:460] Webserver started at http://127.16.43.131:42495/ using document root <none> and password file <none>
I20260620 09:26:28.243636 22463 fs_manager.cc:359] Metadata directory not provided
I20260620 09:26:28.243707 22463 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:26:28.243855 22463 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:26:28.244959 22463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781947562949448-16558-0/minicluster-data/ts-2/data/instance:
uuid: "8b973ae2f0f3435db0f2cabb3be52b2d"
format_stamp: "Formatted at 2026-06-20 09:26:28 on dist-test-slave-706q"
server_key: "99320ab50251e5cfc3ac1f28fcfbea3e"
server_key_iv: "654de19043c6461c4b77958202fa118c"
server_key_version: "encryptionkey@0"
I20260620 09:26:28.245353 22463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781947562949448-16558-0/minicluster-data/ts-2/wal/instance:
uuid: "8b973ae2f0f3435db0f2cabb3be52b2d"
format_stamp: "Formatted at 2026-06-20 09:26:28 on dist-test-slave-706q"
server_key: "99320ab50251e5cfc3ac1f28fcfbea3e"
server_key_iv: "654de19043c6461c4b77958202fa118c"
server_key_version: "encryptionkey@0"
I20260620 09:26:28.246788 22463 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 09:26:28.247772 22478 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:28.247949 22463 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:26:28.248039 22463 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781947562949448-16558-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781947562949448-16558-0/minicluster-data/ts-2/wal
uuid: "8b973ae2f0f3435db0f2cabb3be52b2d"
format_stamp: "Formatted at 2026-06-20 09:26:28 on dist-test-slave-706q"
server_key: "99320ab50251e5cfc3ac1f28fcfbea3e"
server_key_iv: "654de19043c6461c4b77958202fa118c"
server_key_version: "encryptionkey@0"
I20260620 09:26:28.248095 22463 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781947562949448-16558-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781947562949448-16558-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781947562949448-16558-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:26:28.271473 22463 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:26:28.274026 22463 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:26:28.274168 22463 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:26:28.274379 22463 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:26:28.274737 22463 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:26:28.274777 22463 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:28.274802 22463 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:26:28.274816 22463 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:28.281049 22463 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.131:36047
I20260620 09:26:28.281126 22591 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.131:36047 every 8 connection(s)
I20260620 09:26:28.281597 22463 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781947562949448-16558-0/minicluster-data/ts-2/data/info.pb
I20260620 09:26:28.282636 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 22463
I20260620 09:26:28.282745 16558 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781947562949448-16558-0/minicluster-data/ts-2/wal/instance
I20260620 09:26:28.282912 16558 external_mini_cluster.cc:1428] Setting key b318209f287bcfe5e9863502d6d1c014
Jun 20 09:26:28 dist-test-slave-706q krb5kdc[22088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947588, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.131@KRBTEST.COM for kudu/127.16.43.190@KRBTEST.COM
I20260620 09:26:28.293037 22172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:28.282510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.131:51491 (local address 127.16.43.190:37107)
0620 09:26:28.282567 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:28.282571 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:28.282971 (+ 400us) server_negotiation.cc:407] Connection header received
0620 09:26:28.284430 (+ 1459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:28.284431 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:28.284444 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:28.284494 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:28.285220 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:28.285778 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:28.286908 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:28.287132 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:28.287230 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:28.289919 (+ 2689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:28.289921 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:28.289921 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:26:28.289936 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:28.291392 (+ 1456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:28.291914 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:28.291916 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:28.291916 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:28.291978 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:28.292268 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:28.292269 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:28.292269 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:28.292394 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:28.292467 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:28.292846 (+ 379us) server_negotiation.cc:299] Negotiation successful
0620 09:26:28.292933 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 09:26:28.293332 22594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:28.282496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:37107 (local address 127.16.43.131:51491)
0620 09:26:28.282807 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:28.282864 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:28.284278 (+ 1414us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:28.284581 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:28.284586 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:28.284877 (+ 291us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:28.285090 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:28.285096 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:28.285895 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:28.285896 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:28.286797 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:28.286798 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:28.287239 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:28.287240 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:28.287383 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:28.287991 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:26:28.287999 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:26:28.289796 (+ 1797us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:26:28.291511 (+ 1715us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:28.291518 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:28.291520 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:28.291820 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:28.292049 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:28.292050 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:28.292050 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:28.292173 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:28.292472 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:26:28.292479 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:26:28.292624 (+ 145us) client_negotiation.cc:769] Sending connection context
0620 09:26:28.292807 (+ 183us) client_negotiation.cc:240] Negotiation successful
0620 09:26:28.293026 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":103,"threads_started":1}
I20260620 09:26:28.293978 22592 heartbeater.cc:347] Connected to a master server at 127.16.43.190:37107
I20260620 09:26:28.294086 22592 heartbeater.cc:464] Registering TS with master...
I20260620 09:26:28.294301 22592 heartbeater.cc:511] Master 127.16.43.190:37107 requested a full tablet report, sending...
I20260620 09:26:28.294790 22133 ts_manager.cc:194] Registered new tserver with Master: 8b973ae2f0f3435db0f2cabb3be52b2d (127.16.43.131:36047)
I20260620 09:26:28.295210 22133 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.43.131@KRBTEST.COM'} at 127.16.43.131:51491
I20260620 09:26:28.305871 16558 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781947562949448-16558-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781947562949448-16558-0/krb5kdc/test-admin.keytab.
Jun 20 09:26:28 dist-test-slave-706q krb5kdc[22088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947588, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:26:28.321084 16558 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260620 09:26:28.322762 22599 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:26:28.321951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56712 (local address 127.16.43.129:39365)
0620 09:26:28.322112 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:28.322117 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:28.322130 (+ 13us) server_negotiation.cc:407] Connection header received
0620 09:26:28.322156 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:28.322158 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:28.322172 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:28.322250 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:28.322598 (+ 348us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56712: BlockingRecv error: recv got EOF from 127.0.0.1:56712 (error 108)
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":86,"threads_started":1}
W20260620 09:26:28.322824 16558 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781947562949448-16558-0/krb5kdc/krb5cc' of type FILE
Jun 20 09:26:28 dist-test-slave-706q krb5kdc[22088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947588, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:26:28.323871 16558 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 09:26:28 dist-test-slave-706q krb5kdc[22088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947588, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:26:28.324748 16558 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 09:26:28 dist-test-slave-706q krb5kdc[22088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947588, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.43.129@KRBTEST.COM
I20260620 09:26:28.332793 22599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:28.325295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56714 (local address 127.16.43.129:39365)
0620 09:26:28.325364 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:28.325365 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:28.325373 (+ 8us) server_negotiation.cc:407] Connection header received
0620 09:26:28.325464 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:28.325464 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:28.325473 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:28.325523 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:28.325950 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:28.326663 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:28.327357 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:28.327722 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:28.327784 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:28.329969 (+ 2185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:28.329976 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:28.329977 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:26:28.329983 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:28.331679 (+ 1696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:28.332017 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:28.332020 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:28.332021 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:28.332072 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:28.332289 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:28.332290 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:28.332290 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:28.332408 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:28.332507 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:28.332576 (+ 69us) server_negotiation.cc:299] Negotiation successful
0620 09:26:28.332632 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
W20260620 09:26:28.334616 22599 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:26:28.334116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56716 (local address 127.16.43.129:39365)
0620 09:26:28.334151 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:28.334153 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:28.334226 (+ 73us) server_negotiation.cc:407] Connection header received
0620 09:26:28.334270 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:28.334271 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:28.334281 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:28.334325 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:28.334560 (+ 235us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56716: BlockingRecv error: recv got EOF from 127.0.0.1:56716 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260620 09:26:28.334832 16558 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781947562949448-16558-0/krb5kdc/krb5cc' of type FILE
Jun 20 09:26:28 dist-test-slave-706q krb5kdc[22088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947588, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:26:28.335824 16558 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 09:26:28 dist-test-slave-706q krb5kdc[22088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947588, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:26:28.336643 16558 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 09:26:28 dist-test-slave-706q krb5kdc[22088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947588, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.43.129@KRBTEST.COM
I20260620 09:26:28.343169 22599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:28.337098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56722 (local address 127.16.43.129:39365)
0620 09:26:28.337156 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:28.337158 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:28.337181 (+ 23us) server_negotiation.cc:407] Connection header received
0620 09:26:28.337228 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:28.337229 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:28.337236 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:28.337289 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:28.337629 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:28.338110 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:28.338794 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:28.338979 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:28.339027 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:28.341028 (+ 2001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:28.341029 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:28.341030 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:26:28.341035 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:28.342187 (+ 1152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:28.342471 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:28.342471 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:28.342472 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:28.342518 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:28.342766 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:28.342766 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:28.342766 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:28.342847 (+ 81us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:28.342922 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:28.343054 (+ 132us) server_negotiation.cc:299] Negotiation successful
0620 09:26:28.343110 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
W20260620 09:26:28.344864 22599 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:26:28.344354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56724 (local address 127.16.43.129:39365)
0620 09:26:28.344390 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:28.344392 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:28.344433 (+ 41us) server_negotiation.cc:407] Connection header received
0620 09:26:28.344503 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:28.344504 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:28.344513 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:28.344566 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:28.344820 (+ 254us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56724: BlockingRecv error: recv got EOF from 127.0.0.1:56724 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260620 09:26:28.345033 16558 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781947562949448-16558-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781947562949448-16558-0/krb5kdc/krb5cc)
Jun 20 09:26:28 dist-test-slave-706q krb5kdc[22088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947588, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:26:28.345963 16558 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 09:26:28 dist-test-slave-706q krb5kdc[22088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947588, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:26:28.346765 16558 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 09:26:28 dist-test-slave-706q krb5kdc[22088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947588, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.43.129@KRBTEST.COM
I20260620 09:26:28.353713 22599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:28.347225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56732 (local address 127.16.43.129:39365)
0620 09:26:28.347278 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:28.347280 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:28.347288 (+ 8us) server_negotiation.cc:407] Connection header received
0620 09:26:28.347333 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:28.347333 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:28.347340 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:28.347388 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:28.347868 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:28.348318 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:28.349133 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:28.349320 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:28.349374 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:28.351378 (+ 2004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:28.351379 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:28.351379 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:26:28.351385 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:28.352669 (+ 1284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:28.352985 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:28.352986 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:28.352986 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:28.353030 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:28.353206 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:28.353206 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:28.353207 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:28.353299 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:28.353415 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:28.353545 (+ 130us) server_negotiation.cc:299] Negotiation successful
0620 09:26:28.353618 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260620 09:26:28.354411 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 22195
I20260620 09:26:28.358860 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 22329
I20260620 09:26:28.363401 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 22463
I20260620 09:26:28.368324 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 22101
2026-06-20T09:26:28Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1905 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:26:28 dist-test-slave-706q krb5kdc[22612](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:26:28 dist-test-slave-706q krb5kdc[22612](info): set up 2 sockets
Jun 20 09:26:28 dist-test-slave-706q krb5kdc[22612](info): commencing operation
krb5kdc: starting...
W20260620 09:26:29.489048 16558 mini_kdc.cc:121] Time spent starting KDC: real 1.099s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/krb5kdc/test-user.keytab.
Jun 20 09:26:29 dist-test-slave-706q krb5kdc[22612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947589, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:26:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:26:29Z Disabled control of system clock
WARNING: no policy specified for oryx/127.16.43.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.43.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.43.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/krb5kdc/oryx_127.16.43.190.keytab.
Entry for principal oryx/127.16.43.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/krb5kdc/oryx_127.16.43.190.keytab.
I20260620 09:26:29.614714 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.43.190:40995
--webserver_interface=127.16.43.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/krb5kdc/oryx_127.16.43.190.keytab
--principal=oryx/127.16.43.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.43.148:34583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.43.190:40995
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:26:29.697271 22625 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:26:29.697495 22625 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:26:29.697520 22625 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:26:29.698745 22625 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:26:29.698796 22625 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:26:29.698810 22625 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:26:29.698823 22625 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:26:29.698835 22625 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:26:29.700449 22625 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:34583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.43.190:40995
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/krb5kdc/oryx_127.16.43.190.keytab
--principal=oryx/127.16.43.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.43.190:40995
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.43.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22625
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:26:29.700726 22625 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:26:29.700975 22625 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:26:29.706317 22633 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:26:29.706415 22631 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:26:29.706322 22630 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:26:29.706358 22625 server_base.cc:1034] running on GCE node
I20260620 09:26:29.706781 22625 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:26:29.707043 22625 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:26:29.708205 22625 hybrid_clock.cc:648] HybridClock initialized: now 1781947589708180 us; error 47 us; skew 500 ppm
Jun 20 09:26:29 dist-test-slave-706q krb5kdc[22612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947589, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.43.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:26:29.710999 22625 init.cc:377] Logged in from keytab as oryx/127.16.43.190@KRBTEST.COM (short username oryx)
I20260620 09:26:29.711450 22625 webserver.cc:460] Webserver started at http://127.16.43.190:41197/ using document root <none> and password file <none>
I20260620 09:26:29.711726 22625 fs_manager.cc:359] Metadata directory not provided
I20260620 09:26:29.711782 22625 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:26:29.711915 22625 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:26:29.713071 22625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/master-0/data/instance:
uuid: "1d2095405b6a4278a42cc71c3c787e3e"
format_stamp: "Formatted at 2026-06-20 09:26:29 on dist-test-slave-706q"
server_key: "1eff380b188d8d8514e5791be9c12fb6"
server_key_iv: "6231f4e59f2606ca23387f0718e91304"
server_key_version: "encryptionkey@0"
I20260620 09:26:29.713435 22625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/master-0/wal/instance:
uuid: "1d2095405b6a4278a42cc71c3c787e3e"
format_stamp: "Formatted at 2026-06-20 09:26:29 on dist-test-slave-706q"
server_key: "1eff380b188d8d8514e5791be9c12fb6"
server_key_iv: "6231f4e59f2606ca23387f0718e91304"
server_key_version: "encryptionkey@0"
I20260620 09:26:29.714901 22625 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 09:26:29.715839 22640 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:29.716066 22625 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:26:29.716176 22625 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/master-0/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/master-0/wal
uuid: "1d2095405b6a4278a42cc71c3c787e3e"
format_stamp: "Formatted at 2026-06-20 09:26:29 on dist-test-slave-706q"
server_key: "1eff380b188d8d8514e5791be9c12fb6"
server_key_iv: "6231f4e59f2606ca23387f0718e91304"
server_key_version: "encryptionkey@0"
I20260620 09:26:29.716284 22625 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:26:29.736868 22625 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:26:29.738894 22625 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:26:29.739063 22625 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:26:29.743206 22625 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.190:40995
I20260620 09:26:29.743250 22692 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.190:40995 every 8 connection(s)
I20260620 09:26:29.743722 22625 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
I20260620 09:26:29.744787 22693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:26:29.747223 22693 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d2095405b6a4278a42cc71c3c787e3e: Bootstrap starting.
I20260620 09:26:29.747869 22693 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1d2095405b6a4278a42cc71c3c787e3e: Neither blocks nor log segments found. Creating new log.
I20260620 09:26:29.748189 22693 log.cc:824] T 00000000000000000000000000000000 P 1d2095405b6a4278a42cc71c3c787e3e: Log is configured to *not* fsync() on all Append() calls
I20260620 09:26:29.749001 22693 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d2095405b6a4278a42cc71c3c787e3e: No bootstrap required, opened a new log
I20260620 09:26:29.749914 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 22625
I20260620 09:26:29.750007 16558 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/master-0/wal/instance
I20260620 09:26:29.750227 16558 external_mini_cluster.cc:1428] Setting key 34d5122132a7a7af3ecf5331c3eb059c
I20260620 09:26:29.751583 22693 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1d2095405b6a4278a42cc71c3c787e3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d2095405b6a4278a42cc71c3c787e3e" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 40995 } }
I20260620 09:26:29.751755 22693 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1d2095405b6a4278a42cc71c3c787e3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:26:29.751802 22693 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1d2095405b6a4278a42cc71c3c787e3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d2095405b6a4278a42cc71c3c787e3e, State: Initialized, Role: FOLLOWER
I20260620 09:26:29.751958 22693 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d2095405b6a4278a42cc71c3c787e3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d2095405b6a4278a42cc71c3c787e3e" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 40995 } }
I20260620 09:26:29.752048 22693 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1d2095405b6a4278a42cc71c3c787e3e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:26:29.752101 22693 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1d2095405b6a4278a42cc71c3c787e3e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:26:29.752158 22693 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1d2095405b6a4278a42cc71c3c787e3e [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:26:29.753168 22693 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1d2095405b6a4278a42cc71c3c787e3e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d2095405b6a4278a42cc71c3c787e3e" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 40995 } }
I20260620 09:26:29.753294 22693 leader_election.cc:302] T 00000000000000000000000000000000 P 1d2095405b6a4278a42cc71c3c787e3e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1d2095405b6a4278a42cc71c3c787e3e; no voters:
I20260620 09:26:29.753523 22693 leader_election.cc:288] T 00000000000000000000000000000000 P 1d2095405b6a4278a42cc71c3c787e3e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:26:29.753551 22698 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1d2095405b6a4278a42cc71c3c787e3e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:26:29.753672 22698 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1d2095405b6a4278a42cc71c3c787e3e [term 1 LEADER]: Becoming Leader. State: Replica: 1d2095405b6a4278a42cc71c3c787e3e, State: Running, Role: LEADER
I20260620 09:26:29.753760 22698 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1d2095405b6a4278a42cc71c3c787e3e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d2095405b6a4278a42cc71c3c787e3e" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 40995 } }
I20260620 09:26:29.753813 22693 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1d2095405b6a4278a42cc71c3c787e3e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:26:29.754087 22699 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d2095405b6a4278a42cc71c3c787e3e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1d2095405b6a4278a42cc71c3c787e3e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d2095405b6a4278a42cc71c3c787e3e" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 40995 } } }
I20260620 09:26:29.754179 22699 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d2095405b6a4278a42cc71c3c787e3e [sys.catalog]: This master's current role is: LEADER
I20260620 09:26:29.754330 22700 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d2095405b6a4278a42cc71c3c787e3e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d2095405b6a4278a42cc71c3c787e3e. Latest consensus state: current_term: 1 leader_uuid: "1d2095405b6a4278a42cc71c3c787e3e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d2095405b6a4278a42cc71c3c787e3e" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 40995 } } }
I20260620 09:26:29.754415 22700 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d2095405b6a4278a42cc71c3c787e3e [sys.catalog]: This master's current role is: LEADER
I20260620 09:26:29.754408 22703 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:26:29.755189 22703 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:26:29.757212 22703 catalog_manager.cc:1347] Generated new cluster ID: 76574a0d1fa54743b9e218ccdfafe710
I20260620 09:26:29.757268 22703 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 09:26:29 dist-test-slave-706q krb5kdc[22612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947589, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.43.190@KRBTEST.COM
I20260620 09:26:29.761782 22697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:29.750830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40526 (local address 127.16.43.190:40995)
0620 09:26:29.751001 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:29.751009 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:29.751035 (+ 26us) server_negotiation.cc:407] Connection header received
0620 09:26:29.751580 (+ 545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:29.751595 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:29.751651 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:29.751820 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:29.752679 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:29.753871 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:29.755011 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:29.755583 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:29.756152 (+ 569us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:29.758372 (+ 2220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:29.758376 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:29.758376 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:26:29.758388 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:29.760208 (+ 1820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:29.760587 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:29.760590 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:29.760590 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:29.760706 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:29.760935 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:29.760936 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:29.760937 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:29.761178 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:29.761351 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:29.761423 (+ 72us) server_negotiation.cc:299] Negotiation successful
0620 09:26:29.761568 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":90,"threads_started":1}
I20260620 09:26:29.781782 22703 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:26:29.782557 22703 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:26:29.797930 22703 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1d2095405b6a4278a42cc71c3c787e3e: Generated new TSK 0
I20260620 09:26:29.798177 22703 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.16.43.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.43.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.43.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/krb5kdc/oryx_127.16.43.129.keytab.
Entry for principal oryx/127.16.43.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/krb5kdc/oryx_127.16.43.129.keytab.
I20260620 09:26:29.827185 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.43.129:0
--local_ip_for_outbound_sockets=127.16.43.129
--webserver_interface=127.16.43.129
--webserver_port=0
--tserver_master_addrs=127.16.43.190:40995
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/krb5kdc/oryx_127.16.43.129.keytab
--principal=oryx/127.16.43.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.43.148:34583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:26:29.931151 22719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:26:29.931391 22719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:26:29.931416 22719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:26:29.932760 22719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:26:29.932806 22719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:26:29.932835 22719 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.43.129
I20260620 09:26:29.934293 22719 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:34583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/krb5kdc/oryx_127.16.43.129.keytab
--principal=oryx/127.16.43.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.43.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.43.129
--webserver_port=0
--tserver_master_addrs=127.16.43.190:40995
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22719
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.43.129
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:26:29.934494 22719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:26:29.934687 22719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:26:29.939991 22727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:26:29.940001 22724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:26:29.940083 22725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:26:29.940119 22719 server_base.cc:1034] running on GCE node
I20260620 09:26:29.940474 22719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:26:29.940732 22719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:26:29.941900 22719 hybrid_clock.cc:648] HybridClock initialized: now 1781947589941872 us; error 52 us; skew 500 ppm
Jun 20 09:26:29 dist-test-slave-706q krb5kdc[22612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947589, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.43.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:26:29.944666 22719 init.cc:377] Logged in from keytab as oryx/127.16.43.129@KRBTEST.COM (short username oryx)
I20260620 09:26:29.945128 22719 webserver.cc:460] Webserver started at http://127.16.43.129:42355/ using document root <none> and password file <none>
I20260620 09:26:29.945374 22719 fs_manager.cc:359] Metadata directory not provided
I20260620 09:26:29.945420 22719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:26:29.945575 22719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:26:29.946640 22719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/ts-0/data/instance:
uuid: "625889083e23420fb414432cf88b0b51"
format_stamp: "Formatted at 2026-06-20 09:26:29 on dist-test-slave-706q"
server_key: "f0a471b8319018ffbbd50c4f0fc4e1c5"
server_key_iv: "73090e7bc29894b768ea31c99f082bd2"
server_key_version: "encryptionkey@0"
I20260620 09:26:29.947001 22719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/ts-0/wal/instance:
uuid: "625889083e23420fb414432cf88b0b51"
format_stamp: "Formatted at 2026-06-20 09:26:29 on dist-test-slave-706q"
server_key: "f0a471b8319018ffbbd50c4f0fc4e1c5"
server_key_iv: "73090e7bc29894b768ea31c99f082bd2"
server_key_version: "encryptionkey@0"
I20260620 09:26:29.948457 22719 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 09:26:29.949363 22734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:29.949596 22719 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 09:26:29.949726 22719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/ts-0/wal
uuid: "625889083e23420fb414432cf88b0b51"
format_stamp: "Formatted at 2026-06-20 09:26:29 on dist-test-slave-706q"
server_key: "f0a471b8319018ffbbd50c4f0fc4e1c5"
server_key_iv: "73090e7bc29894b768ea31c99f082bd2"
server_key_version: "encryptionkey@0"
I20260620 09:26:29.949810 22719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:26:29.969731 22719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:26:29.972216 22719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:26:29.972389 22719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:26:29.972635 22719 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:26:29.972950 22719 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:26:29.973006 22719 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:29.973058 22719 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:26:29.973088 22719 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:29.978767 22719 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.129:37091
I20260620 09:26:29.978868 22847 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.129:37091 every 8 connection(s)
I20260620 09:26:29.979193 22719 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/ts-0/data/info.pb
I20260620 09:26:29.983489 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 22719
I20260620 09:26:29.983709 16558 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/ts-0/wal/instance
I20260620 09:26:29.983963 16558 external_mini_cluster.cc:1428] Setting key da8e5b921bba32d591ff266525eecbef
Jun 20 09:26:29 dist-test-slave-706q krb5kdc[22612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947589, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.43.129@KRBTEST.COM for oryx/127.16.43.190@KRBTEST.COM
I20260620 09:26:29.989571 22697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:29.980005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.129:58319 (local address 127.16.43.190:40995)
0620 09:26:29.980078 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:29.980080 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:29.980291 (+ 211us) server_negotiation.cc:407] Connection header received
0620 09:26:29.980784 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:29.980785 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:29.980798 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:29.980857 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:29.981864 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:29.982367 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:29.983458 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:29.983828 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:29.983896 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:29.986290 (+ 2394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:29.986291 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:29.986292 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:26:29.986298 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:29.987903 (+ 1605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:29.988461 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:29.988462 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:29.988462 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:29.988516 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:29.988840 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:29.988841 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:29.988842 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:29.988985 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:29.989049 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:29.989423 (+ 374us) server_negotiation.cc:299] Negotiation successful
0620 09:26:29.989502 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 09:26:29.990214 22850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:29.979962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:40995 (local address 127.16.43.129:58319)
0620 09:26:29.980151 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:29.980198 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:29.980702 (+ 504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:29.980915 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:29.980918 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:29.981555 (+ 637us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:29.981765 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:29.981770 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:29.982475 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:29.982476 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:29.983367 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:29.983370 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:29.983912 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:29.983914 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:29.984071 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:29.984642 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:26:29.984652 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:26:29.986183 (+ 1531us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:26:29.987993 (+ 1810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:29.988001 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:29.988003 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:29.988372 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:29.988600 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:29.988602 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:29.988602 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:29.988748 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:29.989057 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:26:29.989065 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:26:29.989252 (+ 187us) client_negotiation.cc:769] Sending connection context
0620 09:26:29.989385 (+ 133us) client_negotiation.cc:240] Negotiation successful
0620 09:26:29.989582 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":91,"threads_started":1}
I20260620 09:26:29.990797 22848 heartbeater.cc:347] Connected to a master server at 127.16.43.190:40995
I20260620 09:26:29.990911 22848 heartbeater.cc:464] Registering TS with master...
I20260620 09:26:29.991110 22848 heartbeater.cc:511] Master 127.16.43.190:40995 requested a full tablet report, sending...
I20260620 09:26:29.991762 22657 ts_manager.cc:194] Registered new tserver with Master: 625889083e23420fb414432cf88b0b51 (127.16.43.129:37091)
I20260620 09:26:29.992401 22657 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.43.129@KRBTEST.COM'} at 127.16.43.129:58319
WARNING: no policy specified for oryx/127.16.43.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.43.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.43.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/krb5kdc/oryx_127.16.43.130.keytab.
Entry for principal oryx/127.16.43.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/krb5kdc/oryx_127.16.43.130.keytab.
I20260620 09:26:30.016798 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.43.130:0
--local_ip_for_outbound_sockets=127.16.43.130
--webserver_interface=127.16.43.130
--webserver_port=0
--tserver_master_addrs=127.16.43.190:40995
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/krb5kdc/oryx_127.16.43.130.keytab
--principal=oryx/127.16.43.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.43.148:34583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:26:30.102398 22853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:26:30.102670 22853 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:26:30.102713 22853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:26:30.104002 22853 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:26:30.104045 22853 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:26:30.104074 22853 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.43.130
I20260620 09:26:30.105504 22853 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:34583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/krb5kdc/oryx_127.16.43.130.keytab
--principal=oryx/127.16.43.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.43.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.43.130
--webserver_port=0
--tserver_master_addrs=127.16.43.190:40995
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22853
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.43.130
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:26:30.105700 22853 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:26:30.105878 22853 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:26:30.111186 22861 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:26:30.111194 22858 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:26:30.111286 22859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:26:30.111459 22853 server_base.cc:1034] running on GCE node
I20260620 09:26:30.111759 22853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:26:30.112037 22853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:26:30.113219 22853 hybrid_clock.cc:648] HybridClock initialized: now 1781947590113197 us; error 41 us; skew 500 ppm
Jun 20 09:26:30 dist-test-slave-706q krb5kdc[22612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947590, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.43.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:26:30.116199 22853 init.cc:377] Logged in from keytab as oryx/127.16.43.130@KRBTEST.COM (short username oryx)
I20260620 09:26:30.116747 22853 webserver.cc:460] Webserver started at http://127.16.43.130:33639/ using document root <none> and password file <none>
I20260620 09:26:30.117010 22853 fs_manager.cc:359] Metadata directory not provided
I20260620 09:26:30.117060 22853 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:26:30.117221 22853 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:26:30.118340 22853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/ts-1/data/instance:
uuid: "318c1dc31882444aa5becb0236f255ea"
format_stamp: "Formatted at 2026-06-20 09:26:30 on dist-test-slave-706q"
server_key: "5a51092e1d76c722d7f6ebe5a5aa544d"
server_key_iv: "69e58789845253ae5677de96d2b3b663"
server_key_version: "encryptionkey@0"
I20260620 09:26:30.118748 22853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/ts-1/wal/instance:
uuid: "318c1dc31882444aa5becb0236f255ea"
format_stamp: "Formatted at 2026-06-20 09:26:30 on dist-test-slave-706q"
server_key: "5a51092e1d76c722d7f6ebe5a5aa544d"
server_key_iv: "69e58789845253ae5677de96d2b3b663"
server_key_version: "encryptionkey@0"
I20260620 09:26:30.120386 22853 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 09:26:30.121366 22868 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:30.121570 22853 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 09:26:30.121655 22853 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/ts-1/wal
uuid: "318c1dc31882444aa5becb0236f255ea"
format_stamp: "Formatted at 2026-06-20 09:26:30 on dist-test-slave-706q"
server_key: "5a51092e1d76c722d7f6ebe5a5aa544d"
server_key_iv: "69e58789845253ae5677de96d2b3b663"
server_key_version: "encryptionkey@0"
I20260620 09:26:30.121707 22853 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:26:30.136850 22853 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:26:30.138795 22853 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:26:30.138904 22853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:26:30.139094 22853 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:26:30.139429 22853 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:26:30.139463 22853 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:30.139487 22853 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:26:30.139554 22853 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:30.145534 22853 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.130:38265
I20260620 09:26:30.145588 22981 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.130:38265 every 8 connection(s)
I20260620 09:26:30.145917 22853 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/ts-1/data/info.pb
Jun 20 09:26:30 dist-test-slave-706q krb5kdc[22612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947590, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.43.130@KRBTEST.COM for oryx/127.16.43.190@KRBTEST.COM
I20260620 09:26:30.153220 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 22853
I20260620 09:26:30.153333 16558 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/ts-1/wal/instance
I20260620 09:26:30.153558 16558 external_mini_cluster.cc:1428] Setting key 707b2304375ced08fddcc1cf8f807e67
I20260620 09:26:30.156328 22697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:30.146703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.130:59747 (local address 127.16.43.190:40995)
0620 09:26:30.146784 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:30.146786 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:30.146980 (+ 194us) server_negotiation.cc:407] Connection header received
0620 09:26:30.147496 (+ 516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:30.147497 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:30.147508 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:30.147619 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:30.148575 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:30.149149 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:30.150160 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:30.150352 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:30.150414 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:30.152631 (+ 2217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:30.152633 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:30.152633 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:26:30.152639 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:30.154676 (+ 2037us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:30.155193 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:30.155194 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:30.155195 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:30.155267 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:30.155522 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:30.155523 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:30.155547 (+ 24us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:30.155692 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:30.155770 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:30.156111 (+ 341us) server_negotiation.cc:299] Negotiation successful
0620 09:26:30.156208 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 09:26:30.156394 22984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:30.146673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:40995 (local address 127.16.43.130:59747)
0620 09:26:30.146876 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:30.146913 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:30.147407 (+ 494us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:30.147682 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:30.147685 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:30.148259 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:30.148461 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:30.148466 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:30.149259 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:30.149259 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:30.150061 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:30.150063 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:30.150432 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:30.150432 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:30.150533 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:30.151059 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:26:30.151067 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:26:30.152539 (+ 1472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:26:30.154762 (+ 2223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:30.154789 (+ 27us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:30.154792 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:30.155089 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:30.155338 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:30.155339 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:30.155339 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:30.155453 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:30.155774 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:26:30.155783 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:26:30.155967 (+ 184us) client_negotiation.cc:769] Sending connection context
0620 09:26:30.156063 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 09:26:30.156191 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":132,"threads_started":1}
I20260620 09:26:30.156888 22982 heartbeater.cc:347] Connected to a master server at 127.16.43.190:40995
I20260620 09:26:30.157002 22982 heartbeater.cc:464] Registering TS with master...
I20260620 09:26:30.157186 22982 heartbeater.cc:511] Master 127.16.43.190:40995 requested a full tablet report, sending...
I20260620 09:26:30.157536 22657 ts_manager.cc:194] Registered new tserver with Master: 318c1dc31882444aa5becb0236f255ea (127.16.43.130:38265)
I20260620 09:26:30.157927 22657 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.43.130@KRBTEST.COM'} at 127.16.43.130:59747
WARNING: no policy specified for oryx/127.16.43.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.43.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.43.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/krb5kdc/oryx_127.16.43.131.keytab.
Entry for principal oryx/127.16.43.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/krb5kdc/oryx_127.16.43.131.keytab.
I20260620 09:26:30.178967 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.43.131:0
--local_ip_for_outbound_sockets=127.16.43.131
--webserver_interface=127.16.43.131
--webserver_port=0
--tserver_master_addrs=127.16.43.190:40995
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/krb5kdc/oryx_127.16.43.131.keytab
--principal=oryx/127.16.43.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.43.148:34583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:26:30.260223 22987 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:26:30.260473 22987 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:26:30.260511 22987 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:26:30.261709 22987 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:26:30.261780 22987 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:26:30.261824 22987 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.43.131
I20260620 09:26:30.263306 22987 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:34583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/krb5kdc/oryx_127.16.43.131.keytab
--principal=oryx/127.16.43.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.43.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.43.131
--webserver_port=0
--tserver_master_addrs=127.16.43.190:40995
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22987
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.43.131
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:26:30.263631 22987 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:26:30.263891 22987 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:26:30.269498 22992 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:26:30.269562 22995 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:26:30.269586 22993 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:26:30.269821 22987 server_base.cc:1034] running on GCE node
I20260620 09:26:30.270032 22987 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:26:30.270236 22987 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:26:30.271379 22987 hybrid_clock.cc:648] HybridClock initialized: now 1781947590271368 us; error 29 us; skew 500 ppm
Jun 20 09:26:30 dist-test-slave-706q krb5kdc[22612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947590, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.43.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:26:30.274263 22987 init.cc:377] Logged in from keytab as oryx/127.16.43.131@KRBTEST.COM (short username oryx)
I20260620 09:26:30.274796 22987 webserver.cc:460] Webserver started at http://127.16.43.131:38635/ using document root <none> and password file <none>
I20260620 09:26:30.275046 22987 fs_manager.cc:359] Metadata directory not provided
I20260620 09:26:30.275094 22987 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:26:30.275230 22987 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:26:30.276398 22987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/ts-2/data/instance:
uuid: "3a91f37200fc4719b1d4853f3fbaefe3"
format_stamp: "Formatted at 2026-06-20 09:26:30 on dist-test-slave-706q"
server_key: "8a32481ced996d71f396f6feda360e65"
server_key_iv: "61a1d007146cbede6f3521b91b4bb174"
server_key_version: "encryptionkey@0"
I20260620 09:26:30.276758 22987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/ts-2/wal/instance:
uuid: "3a91f37200fc4719b1d4853f3fbaefe3"
format_stamp: "Formatted at 2026-06-20 09:26:30 on dist-test-slave-706q"
server_key: "8a32481ced996d71f396f6feda360e65"
server_key_iv: "61a1d007146cbede6f3521b91b4bb174"
server_key_version: "encryptionkey@0"
I20260620 09:26:30.278117 22987 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 09:26:30.278867 23002 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:30.279081 22987 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 09:26:30.279191 22987 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/ts-2/wal
uuid: "3a91f37200fc4719b1d4853f3fbaefe3"
format_stamp: "Formatted at 2026-06-20 09:26:30 on dist-test-slave-706q"
server_key: "8a32481ced996d71f396f6feda360e65"
server_key_iv: "61a1d007146cbede6f3521b91b4bb174"
server_key_version: "encryptionkey@0"
I20260620 09:26:30.279273 22987 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:26:30.301586 22987 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:26:30.304076 22987 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:26:30.304248 22987 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:26:30.304503 22987 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:26:30.304829 22987 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:26:30.304862 22987 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:30.304885 22987 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:26:30.304934 22987 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:30.310777 22987 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.131:39089
I20260620 09:26:30.310863 23115 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.131:39089 every 8 connection(s)
I20260620 09:26:30.311241 22987 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/ts-2/data/info.pb
I20260620 09:26:30.313973 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 22987
I20260620 09:26:30.314047 16558 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/ts-2/wal/instance
I20260620 09:26:30.314251 16558 external_mini_cluster.cc:1428] Setting key a0186236c7b3475bd9bcdcd4f01c244f
Jun 20 09:26:30 dist-test-slave-706q krb5kdc[22612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947590, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.43.131@KRBTEST.COM for oryx/127.16.43.190@KRBTEST.COM
I20260620 09:26:30.320875 22697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:30.312034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.131:54073 (local address 127.16.43.190:40995)
0620 09:26:30.312139 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:30.312141 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:30.312339 (+ 198us) server_negotiation.cc:407] Connection header received
0620 09:26:30.312808 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:30.312809 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:30.312819 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:30.312872 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:30.313807 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:30.314425 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:30.315779 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:30.315965 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:30.316030 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:30.318183 (+ 2153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:30.318185 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:30.318185 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:26:30.318192 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:30.319648 (+ 1456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:30.320047 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:30.320048 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:30.320048 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:30.320098 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:30.320307 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:30.320308 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:30.320308 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:30.320410 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:30.320461 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:30.320751 (+ 290us) server_negotiation.cc:299] Negotiation successful
0620 09:26:30.320811 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 09:26:30.321076 23118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:30.312002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:40995 (local address 127.16.43.131:54073)
0620 09:26:30.312208 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:30.312245 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:30.312719 (+ 474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:30.312928 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:30.312931 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:30.313526 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:30.313726 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:30.313730 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:30.314582 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:30.314583 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:30.315697 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:30.315700 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:30.316040 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:30.316040 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:30.316160 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:30.316633 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:26:30.316642 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:26:30.318101 (+ 1459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:26:30.319750 (+ 1649us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:30.319755 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:30.319757 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:30.319979 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:30.320155 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:30.320156 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:30.320156 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:30.320251 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:30.320464 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:26:30.320468 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:26:30.320580 (+ 112us) client_negotiation.cc:769] Sending connection context
0620 09:26:30.320724 (+ 144us) client_negotiation.cc:240] Negotiation successful
0620 09:26:30.320846 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":106,"threads_started":1}
I20260620 09:26:30.321583 23116 heartbeater.cc:347] Connected to a master server at 127.16.43.190:40995
I20260620 09:26:30.321695 23116 heartbeater.cc:464] Registering TS with master...
I20260620 09:26:30.321916 23116 heartbeater.cc:511] Master 127.16.43.190:40995 requested a full tablet report, sending...
I20260620 09:26:30.322371 22657 ts_manager.cc:194] Registered new tserver with Master: 3a91f37200fc4719b1d4853f3fbaefe3 (127.16.43.131:39089)
I20260620 09:26:30.322923 22657 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.43.131@KRBTEST.COM'} at 127.16.43.131:54073
I20260620 09:26:30.326196 16558 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.16.43.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.43.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.43.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/krb5kdc/oryx_127.16.43.189.keytab.
Entry for principal oryx/127.16.43.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/krb5kdc/oryx_127.16.43.189.keytab.
I20260620 09:26:30.354921 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.43.189:33891
--webserver_interface=127.16.43.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/krb5kdc/oryx_127.16.43.189.keytab
--principal=oryx/127.16.43.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.43.148:34583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.43.190:40995,127.16.43.189:33891
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:26:30.440765 23121 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:26:30.441018 23121 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:26:30.441056 23121 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:26:30.442281 23121 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:26:30.442356 23121 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:26:30.442381 23121 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:26:30.442409 23121 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:26:30.442441 23121 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:26:30.443951 23121 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:34583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.16.43.190:40995,127.16.43.189:33891
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/krb5kdc/oryx_127.16.43.189.keytab
--principal=oryx/127.16.43.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.43.189:33891
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.16.43.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23121
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
W20260620 09:26:30.444096 23121 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.16.43.190:40995,127.16.43.189:33891'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260620 09:26:30.444263 23121 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:26:30.444502 23121 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:26:30.449668 23127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:26:30.449668 23129 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:26:30.449893 23126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:26:30.449921 23121 server_base.cc:1034] running on GCE node
I20260620 09:26:30.450189 23121 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:26:30.450513 23121 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:26:30.451663 23121 hybrid_clock.cc:648] HybridClock initialized: now 1781947590451645 us; error 31 us; skew 500 ppm
Jun 20 09:26:30 dist-test-slave-706q krb5kdc[22612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947590, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.43.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:26:30.454457 23121 init.cc:377] Logged in from keytab as oryx/127.16.43.189@KRBTEST.COM (short username oryx)
I20260620 09:26:30.454926 23121 webserver.cc:460] Webserver started at http://127.16.43.189:40161/ using document root <none> and password file <none>
I20260620 09:26:30.455188 23121 fs_manager.cc:359] Metadata directory not provided
I20260620 09:26:30.455266 23121 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:26:30.455413 23121 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:26:30.456563 23121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/master-1/data/instance:
uuid: "13835b763e164a3f9232bac524dd0bac"
format_stamp: "Formatted at 2026-06-20 09:26:30 on dist-test-slave-706q"
server_key: "4188e02e728e41d6ec365055ce96829e"
server_key_iv: "12c1ac783b9035e67280606a822f6080"
server_key_version: "encryptionkey@0"
I20260620 09:26:30.456934 23121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/master-1/wal/instance:
uuid: "13835b763e164a3f9232bac524dd0bac"
format_stamp: "Formatted at 2026-06-20 09:26:30 on dist-test-slave-706q"
server_key: "4188e02e728e41d6ec365055ce96829e"
server_key_iv: "12c1ac783b9035e67280606a822f6080"
server_key_version: "encryptionkey@0"
I20260620 09:26:30.458320 23121 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 09:26:30.459156 23136 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:30.459333 23121 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 09:26:30.459434 23121 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/master-1/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/master-1/wal
uuid: "13835b763e164a3f9232bac524dd0bac"
format_stamp: "Formatted at 2026-06-20 09:26:30 on dist-test-slave-706q"
server_key: "4188e02e728e41d6ec365055ce96829e"
server_key_iv: "12c1ac783b9035e67280606a822f6080"
server_key_version: "encryptionkey@0"
I20260620 09:26:30.459519 23121 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:26:30.482000 23121 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:26:30.484099 23121 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:26:30.484274 23121 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:26:30.488237 23121 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.189:33891
I20260620 09:26:30.488288 23188 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.189:33891 every 8 connection(s)
I20260620 09:26:30.488673 23121 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/master-1/data/info.pb
I20260620 09:26:30.489703 23189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:26:30.490332 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 23121
I20260620 09:26:30.491729 23189 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.43.190" port: 40995 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 09:26:30 dist-test-slave-706q krb5kdc[22612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947590, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.43.189@KRBTEST.COM for oryx/127.16.43.190@KRBTEST.COM
Jun 20 09:26:30 dist-test-slave-706q krb5kdc[22612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947589, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.43.189@KRBTEST.COM
I20260620 09:26:30.500927 22697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:30.492421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40532 (local address 127.16.43.190:40995)
0620 09:26:30.492487 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:30.492490 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:30.492671 (+ 181us) server_negotiation.cc:407] Connection header received
0620 09:26:30.492700 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:30.492701 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:30.492711 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:30.492753 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:30.493328 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:30.493888 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:30.494879 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:30.495124 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:30.495208 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:30.497485 (+ 2277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:30.497486 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:30.497487 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:26:30.497495 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:30.499352 (+ 1857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:30.499855 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:30.499856 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:30.499857 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:30.499920 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:30.500143 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:30.500144 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:30.500144 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:30.500281 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:30.500336 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:30.500732 (+ 396us) server_negotiation.cc:299] Negotiation successful
0620 09:26:30.500814 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 09:26:30.501376 23193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:30.492358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:40995 (local address 127.0.0.1:40532)
0620 09:26:30.492548 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:30.492584 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:30.492652 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:30.492779 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:30.492782 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:30.493126 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:30.493244 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:30.493249 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:30.493984 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:30.493985 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:30.494770 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:30.494772 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:30.495218 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:30.495219 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:30.495269 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:30.495923 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:26:30.495932 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:26:30.497392 (+ 1460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:26:30.499437 (+ 2045us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:30.499442 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:30.499444 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:30.499781 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:30.499984 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:30.499985 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:30.499985 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:30.500082 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:30.500357 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:26:30.500363 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:26:30.500545 (+ 182us) client_negotiation.cc:769] Sending connection context
0620 09:26:30.500665 (+ 120us) client_negotiation.cc:240] Negotiation successful
0620 09:26:30.500852 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"mutex_wait_us":3,"thread_start_us":91,"threads_started":1}
I20260620 09:26:30.501380 23192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:30.491030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42348 (local address 127.16.43.189:33891)
0620 09:26:30.491195 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:30.491220 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:30.491245 (+ 25us) server_negotiation.cc:407] Connection header received
0620 09:26:30.491695 (+ 450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:30.491703 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:30.491768 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:30.491896 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:30.492634 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:30.493760 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:30.494517 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:30.494959 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:30.495033 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:30.498089 (+ 3056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:30.498096 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:30.498097 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:26:30.498104 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:30.499912 (+ 1808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:30.500426 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:30.500429 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:30.500429 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:30.500487 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:30.500707 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:30.500707 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:30.500708 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:30.500866 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:30.501031 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:30.501155 (+ 124us) server_negotiation.cc:299] Negotiation successful
0620 09:26:30.501212 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":86,"threads_started":1}
I20260620 09:26:30.502094 23189 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.43.189" port: 33891 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 09:26:30 dist-test-slave-706q krb5kdc[22612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947590, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.43.189@KRBTEST.COM for oryx/127.16.43.189@KRBTEST.COM
I20260620 09:26:30.508806 23193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:30.502369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.189:33891 (local address 127.0.0.1:42356)
0620 09:26:30.502418 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:30.502428 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:30.502470 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:30.502563 (+ 93us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:30.502563 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:30.502686 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:30.502731 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:30.502732 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:30.503402 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:30.503403 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:30.503988 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:30.503990 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:30.504373 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:30.504374 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:30.504430 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:30.504915 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:26:30.504919 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:26:30.506157 (+ 1238us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:26:30.507877 (+ 1720us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:30.507878 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:30.507878 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:30.508033 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:30.508239 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:30.508240 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:30.508241 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:30.508299 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:30.508524 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:26:30.508524 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:26:30.508568 (+ 44us) client_negotiation.cc:769] Sending connection context
0620 09:26:30.508620 (+ 52us) client_negotiation.cc:240] Negotiation successful
0620 09:26:30.508703 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260620 09:26:30.508816 23192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:30.502395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42356 (local address 127.16.43.189:33891)
0620 09:26:30.502443 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:30.502445 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:30.502457 (+ 12us) server_negotiation.cc:407] Connection header received
0620 09:26:30.502513 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:30.502514 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:30.502521 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:30.502561 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:30.502778 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:30.503314 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:30.504079 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:30.504289 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:30.504355 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:30.506271 (+ 1916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:30.506273 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:30.506273 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:26:30.506279 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:30.507789 (+ 1510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:30.508099 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:30.508100 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:30.508100 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:30.508159 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:30.508366 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:30.508367 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:30.508367 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:30.508466 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:30.508516 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:30.508642 (+ 126us) server_negotiation.cc:299] Negotiation successful
0620 09:26:30.508747 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":44,"server-negotiator.queue_time_us":30}
I20260620 09:26:30.510195 23189 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13835b763e164a3f9232bac524dd0bac: Bootstrap starting.
I20260620 09:26:30.511039 23189 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 13835b763e164a3f9232bac524dd0bac: Neither blocks nor log segments found. Creating new log.
I20260620 09:26:30.511315 23189 log.cc:824] T 00000000000000000000000000000000 P 13835b763e164a3f9232bac524dd0bac: Log is configured to *not* fsync() on all Append() calls
I20260620 09:26:30.512137 23189 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13835b763e164a3f9232bac524dd0bac: No bootstrap required, opened a new log
I20260620 09:26:30.513562 23189 raft_consensus.cc:348] T 00000000000000000000000000000000 P 13835b763e164a3f9232bac524dd0bac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d2095405b6a4278a42cc71c3c787e3e" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 40995 } } peers { permanent_uuid: "13835b763e164a3f9232bac524dd0bac" member_type: VOTER last_known_addr { host: "127.16.43.189" port: 33891 } }
I20260620 09:26:30.513713 23189 raft_consensus.cc:374] T 00000000000000000000000000000000 P 13835b763e164a3f9232bac524dd0bac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:26:30.513761 23189 raft_consensus.cc:729] T 00000000000000000000000000000000 P 13835b763e164a3f9232bac524dd0bac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13835b763e164a3f9232bac524dd0bac, State: Initialized, Role: FOLLOWER
I20260620 09:26:30.513940 23189 consensus_queue.cc:260] T 00000000000000000000000000000000 P 13835b763e164a3f9232bac524dd0bac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d2095405b6a4278a42cc71c3c787e3e" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 40995 } } peers { permanent_uuid: "13835b763e164a3f9232bac524dd0bac" member_type: VOTER last_known_addr { host: "127.16.43.189" port: 33891 } }
I20260620 09:26:30.514227 23189 sys_catalog.cc:564] T 00000000000000000000000000000000 P 13835b763e164a3f9232bac524dd0bac [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:26:30.514191 23195 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13835b763e164a3f9232bac524dd0bac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d2095405b6a4278a42cc71c3c787e3e" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 40995 } } peers { permanent_uuid: "13835b763e164a3f9232bac524dd0bac" member_type: VOTER last_known_addr { host: "127.16.43.189" port: 33891 } } }
I20260620 09:26:30.514297 23195 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13835b763e164a3f9232bac524dd0bac [sys.catalog]: This master's current role is: FOLLOWER
W20260620 09:26:30.515964 23206 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 13835b763e164a3f9232bac524dd0bac: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 09:26:30.516038 23206 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 09:26:30.517573 23121 master_runner.cc:428] Detected that this master 13835b763e164a3f9232bac524dd0bac is joining an existing cluster
I20260620 09:26:30.517608 23121 master_runner.cc:432] Initiating AddMaster RPC to add 127.16.43.189:33891
I20260620 09:26:30.526367 22697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:30.518761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40534 (local address 127.16.43.190:40995)
0620 09:26:30.518809 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:30.518811 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:30.518947 (+ 136us) server_negotiation.cc:407] Connection header received
0620 09:26:30.519142 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:30.519143 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:30.519154 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:30.519218 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:30.519658 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:30.520471 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:30.521578 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:30.521767 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:30.521858 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:30.523019 (+ 1161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:30.523021 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:30.523022 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:26:30.523029 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:30.524771 (+ 1742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:30.525513 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:30.525515 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:30.525515 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:30.525580 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:30.525799 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:30.525800 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:30.525800 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:30.525944 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:30.526026 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:30.526196 (+ 170us) server_negotiation.cc:299] Negotiation successful
0620 09:26:30.526276 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 09:26:30.526379 23212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:30.518727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:40995 (local address 127.0.0.1:40534)
0620 09:26:30.518884 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:30.518895 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:30.519080 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:30.519220 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:30.519221 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:30.519447 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:30.519577 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:30.519579 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:30.520565 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:30.520566 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:30.521483 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:30.521486 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:30.521857 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:30.521858 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:30.521929 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:30.522442 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:26:30.522447 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:26:30.522932 (+ 485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:26:30.524862 (+ 1930us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:30.524863 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:30.524864 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:30.525433 (+ 569us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:30.525659 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:30.525660 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:30.525660 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:30.525728 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:30.526029 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:26:30.526031 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:26:30.526074 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 09:26:30.526155 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 09:26:30.526261 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":104,"threads_started":1}
I20260620 09:26:30.527215 23213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:30.518965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.189:33891 (local address 127.0.0.1:42370)
0620 09:26:30.519086 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:30.519097 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:30.519178 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:30.519339 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:30.519339 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:30.519660 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:30.519721 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:30.519723 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:30.521586 (+ 1863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:30.521587 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:30.522411 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:30.522413 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:30.522849 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:30.522850 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:30.522916 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:30.523339 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:26:30.523345 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:26:30.523744 (+ 399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:26:30.525900 (+ 2156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:30.525901 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:30.525901 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:30.526098 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:30.526286 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:30.526287 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:30.526287 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:30.526336 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:30.526843 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:26:30.526844 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:26:30.526899 (+ 55us) client_negotiation.cc:769] Sending connection context
0620 09:26:30.526964 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 09:26:30.527065 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":69,"threads_started":1}
I20260620 09:26:30.527230 23192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:30.519025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42370 (local address 127.16.43.189:33891)
0620 09:26:30.519062 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:30.519064 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:30.519156 (+ 92us) server_negotiation.cc:407] Connection header received
0620 09:26:30.519240 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:30.519241 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:30.519249 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:30.519276 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:30.519799 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:30.520486 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:30.522484 (+ 1998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:30.522758 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:30.522831 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:30.524088 (+ 1257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:30.524090 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:30.524090 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:26:30.524097 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:30.525834 (+ 1737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:30.526176 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:30.526177 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:30.526178 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:30.526225 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:30.526400 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:30.526400 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:30.526401 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:30.526507 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:30.526545 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:30.526991 (+ 446us) server_negotiation.cc:299] Negotiation successful
0620 09:26:30.527128 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":58,"server-negotiator.queue_time_us":21}
Jun 20 09:26:30 dist-test-slave-706q krb5kdc[22612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947589, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.43.190@KRBTEST.COM for oryx/127.16.43.189@KRBTEST.COM
I20260620 09:26:30.536373 23192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:30.528927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42380 (local address 127.16.43.189:33891)
0620 09:26:30.528966 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:30.528968 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:30.529116 (+ 148us) server_negotiation.cc:407] Connection header received
0620 09:26:30.529204 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:30.529205 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:30.529215 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:30.529258 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:30.529782 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:30.530212 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:30.531333 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:30.531509 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:30.531631 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:30.533696 (+ 2065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:30.533697 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:30.533697 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:26:30.533747 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:30.535065 (+ 1318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:30.535498 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:30.535499 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:30.535500 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:30.535599 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:30.535815 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:30.535815 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:30.535816 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:30.535929 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:30.536003 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:30.536196 (+ 193us) server_negotiation.cc:299] Negotiation successful
0620 09:26:30.536274 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 09:26:30.536391 23214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:30.528835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.189:33891 (local address 127.0.0.1:42380)
0620 09:26:30.529007 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:30.529051 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:30.529130 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:30.529273 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:30.529276 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:30.529615 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:30.529673 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:30.529676 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:30.530292 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:30.530293 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:30.531247 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:30.531249 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:30.531634 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:30.531635 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:30.531723 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:30.532160 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:26:30.532167 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:26:30.533605 (+ 1438us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:26:30.535191 (+ 1586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:30.535198 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:30.535200 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:30.535406 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:30.535684 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:30.535685 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:30.535685 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:30.535748 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:30.535995 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:26:30.536003 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:26:30.536055 (+ 52us) client_negotiation.cc:769] Sending connection context
0620 09:26:30.536156 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 09:26:30.536286 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":106,"threads_started":1}
I20260620 09:26:30.536983 22657 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.16.43.189:33891: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "13835b763e164a3f9232bac524dd0bac"
member_type: NON_VOTER
last_known_addr {
host: "127.16.43.189"
port: 33891
}
attrs {
promote: true
}
}
dest_uuid: "1d2095405b6a4278a42cc71c3c787e3e"
cas_config_opid_index: -1
I20260620 09:26:30.537317 22657 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1d2095405b6a4278a42cc71c3c787e3e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1d2095405b6a4278a42cc71c3c787e3e" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 40995 } } peers { permanent_uuid: "13835b763e164a3f9232bac524dd0bac" member_type: NON_VOTER last_known_addr { host: "127.16.43.189" port: 33891 } attrs { promote: true } }
I20260620 09:26:30.537853 23216 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1d2095405b6a4278a42cc71c3c787e3e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 13835b763e164a3f9232bac524dd0bac (127.16.43.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1d2095405b6a4278a42cc71c3c787e3e" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 40995 } } peers { permanent_uuid: "13835b763e164a3f9232bac524dd0bac" member_type: NON_VOTER last_known_addr { host: "127.16.43.189" port: 33891 } attrs { promote: true } } }
I20260620 09:26:30.538187 23163 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 13835b763e164a3f9232bac524dd0bac [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:26:30.538870 23216 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.16.43.189:33891
I20260620 09:26:30.538993 23163 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 13835b763e164a3f9232bac524dd0bac [term 1 FOLLOWER]: Refusing update from remote peer 1d2095405b6a4278a42cc71c3c787e3e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260620 09:26:30.539103 23121 master.cc:537] Master@127.16.43.189:33891 shutting down...
I20260620 09:26:30.540340 23216 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1d2095405b6a4278a42cc71c3c787e3e [LEADER]: Connected to new peer: Peer: permanent_uuid: "13835b763e164a3f9232bac524dd0bac" member_type: NON_VOTER last_known_addr { host: "127.16.43.189" port: 33891 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 09:26:30.541309 22643 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260620 09:26:30.541555 23215 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d2095405b6a4278a42cc71c3c787e3e [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "1d2095405b6a4278a42cc71c3c787e3e" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1d2095405b6a4278a42cc71c3c787e3e" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 40995 } } peers { permanent_uuid: "13835b763e164a3f9232bac524dd0bac" member_type: NON_VOTER last_known_addr { host: "127.16.43.189" port: 33891 } attrs { promote: true } } }
I20260620 09:26:30.541648 23215 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d2095405b6a4278a42cc71c3c787e3e [sys.catalog]: This master's current role is: LEADER
I20260620 09:26:30.542172 23121 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 13835b763e164a3f9232bac524dd0bac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:26:30.542264 23121 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 13835b763e164a3f9232bac524dd0bac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:26:30.542312 23121 tablet_replica.cc:330] T 00000000000000000000000000000000 P 13835b763e164a3f9232bac524dd0bac: stopping tablet replica
I20260620 09:26:30.549113 23214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:30.548965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.189:33891 (local address 127.0.0.1:42394)
0620 09:26:30.549020 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:30.549062 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.189:33891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260620 09:26:30.549275 22643 consensus_peers.cc:487] T 00000000000000000000000000000000 P 1d2095405b6a4278a42cc71c3c787e3e -> Peer 13835b763e164a3f9232bac524dd0bac (127.16.43.189:33891): Couldn't send request to peer 13835b763e164a3f9232bac524dd0bac. Status: Network error: Client connection negotiation failed: client connection to 127.16.43.189:33891: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 09:26:30.554965 23121 master.cc:552] Master@127.16.43.189:33891 shutdown complete.
I20260620 09:26:30.555038 23121 master_runner.cc:305] Clearing existing system tablet
I20260620 09:26:30.555181 23121 fs_manager.cc:359] Metadata directory not provided
I20260620 09:26:30.555230 23121 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:26:30.555962 23121 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260620 09:26:30.556460 23220 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:30.556643 23121 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:26:30.556708 23121 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:30.556739 23121 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/master-1/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/master-1/wal
uuid: "13835b763e164a3f9232bac524dd0bac"
format_stamp: "Formatted at 2026-06-20 09:26:30 on dist-test-slave-706q"
server_key: "4188e02e728e41d6ec365055ce96829e"
server_key_iv: "12c1ac783b9035e67280606a822f6080"
server_key_version: "encryptionkey@0"
I20260620 09:26:30.557389 23121 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 13835b763e164a3f9232bac524dd0bac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:26:30.559110 23121 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 13835b763e164a3f9232bac524dd0bac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 09:26:30.559218 23121 log.cc:1196] T 00000000000000000000000000000000 P 13835b763e164a3f9232bac524dd0bac: Deleting WAL directory at /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260620 09:26:30.559408 23121 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 13835b763e164a3f9232bac524dd0bac: Deleting consensus metadata
I20260620 09:26:30.559612 23121 master_runner.cc:315] Copying system tablet from 127.16.43.190:40995
I20260620 09:26:30.560243 23121 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 13835b763e164a3f9232bac524dd0bac: tablet copy: Beginning tablet copy session from remote peer at address 127.16.43.190:40995
I20260620 09:26:30.566555 22697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:30.560625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40548 (local address 127.16.43.190:40995)
0620 09:26:30.560676 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:30.560679 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:30.560847 (+ 168us) server_negotiation.cc:407] Connection header received
0620 09:26:30.560931 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:30.560933 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:30.560943 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:30.560989 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:30.561324 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:30.561884 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:30.562623 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:30.562838 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:30.562917 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:30.563976 (+ 1059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:30.563978 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:30.563978 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:26:30.563986 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:30.565353 (+ 1367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:30.565732 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:30.565733 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:30.565734 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:30.565794 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:30.566001 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:30.566002 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:30.566002 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:30.566127 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:30.566214 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:30.566373 (+ 159us) server_negotiation.cc:299] Negotiation successful
0620 09:26:30.566455 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 09:26:30.566565 23226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:30.560552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:40995 (local address 127.0.0.1:40548)
0620 09:26:30.560770 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:30.560780 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:30.560864 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:30.561003 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:30.561004 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:30.561187 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:30.561243 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:30.561245 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:30.561979 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:30.561980 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:30.562537 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:30.562538 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:30.562915 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:30.562916 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:30.562982 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:30.563504 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:26:30.563509 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:26:30.563866 (+ 357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:26:30.565456 (+ 1590us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:30.565457 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:30.565458 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:30.565637 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:30.565874 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:30.565874 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:30.565875 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:30.565938 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:30.566196 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:26:30.566197 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:26:30.566245 (+ 48us) client_negotiation.cc:769] Sending connection context
0620 09:26:30.566347 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 09:26:30.566456 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":84,"threads_started":1}
I20260620 09:26:30.566867 22677 tablet_copy_service.cc:140] P 1d2095405b6a4278a42cc71c3c787e3e: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 13835b763e164a3f9232bac524dd0bac ({username='oryx', principal='oryx/127.16.43.189@KRBTEST.COM'} at 127.0.0.1:40548)
I20260620 09:26:30.566972 22677 tablet_copy_service.cc:161] P 1d2095405b6a4278a42cc71c3c787e3e: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 13835b763e164a3f9232bac524dd0bac at {username='oryx', principal='oryx/127.16.43.189@KRBTEST.COM'} at 127.0.0.1:40548: session id = 13835b763e164a3f9232bac524dd0bac-00000000000000000000000000000000
I20260620 09:26:30.567716 22677 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 1d2095405b6a4278a42cc71c3c787e3e: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 09:26:30.568378 23121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 09:26:30.569376 23121 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 13835b763e164a3f9232bac524dd0bac: tablet copy: Starting download of 0 data blocks...
I20260620 09:26:30.569533 23121 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 13835b763e164a3f9232bac524dd0bac: tablet copy: Starting download of 1 WAL segments...
I20260620 09:26:30.570402 23121 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 13835b763e164a3f9232bac524dd0bac: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 09:26:30.571218 22677 tablet_copy_service.cc:342] P 1d2095405b6a4278a42cc71c3c787e3e: Request end of tablet copy session 13835b763e164a3f9232bac524dd0bac-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.16.43.189@KRBTEST.COM'} at 127.0.0.1:40548
I20260620 09:26:30.571275 22677 tablet_copy_service.cc:433] P 1d2095405b6a4278a42cc71c3c787e3e: ending tablet copy session 13835b763e164a3f9232bac524dd0bac-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 13835b763e164a3f9232bac524dd0bac
I20260620 09:26:30.573463 23121 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:26:30.574621 23227 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:26:30.574923 23121 server_base.cc:1034] running on GCE node
W20260620 09:26:30.580534 23230 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:26:30.580531 23228 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:26:30.580849 23121 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:26:30.581039 23121 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:26:30.582170 23121 hybrid_clock.cc:648] HybridClock initialized: now 1781947590582162 us; error 27 us; skew 500 ppm
Jun 20 09:26:30 dist-test-slave-706q krb5kdc[22612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947590, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.43.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:26:30.583675 23121 init.cc:377] Logged in from keytab as oryx/127.16.43.189@KRBTEST.COM (short username oryx)
I20260620 09:26:30.583937 23121 webserver.cc:460] Webserver started at http://127.16.43.189:37589/ using document root <none> and password file <none>
I20260620 09:26:30.584057 23121 fs_manager.cc:359] Metadata directory not provided
I20260620 09:26:30.584093 23121 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:26:30.584805 23121 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260620 09:26:30.585350 23237 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:30.585553 23121 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 09:26:30.585605 23121 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/master-1/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/master-1/wal
uuid: "13835b763e164a3f9232bac524dd0bac"
format_stamp: "Formatted at 2026-06-20 09:26:30 on dist-test-slave-706q"
server_key: "4188e02e728e41d6ec365055ce96829e"
server_key_iv: "12c1ac783b9035e67280606a822f6080"
server_key_version: "encryptionkey@0"
I20260620 09:26:30.585655 23121 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:26:30.608345 23121 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:26:30.610044 23121 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:26:30.613055 23121 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.189:33891
I20260620 09:26:30.613070 23289 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.189:33891 every 8 connection(s)
I20260620 09:26:30.613420 23121 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/master-1/data/info.pb
I20260620 09:26:30.614301 23290 sys_catalog.cc:263] Verifying existing consensus state
I20260620 09:26:30.614858 23290 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13835b763e164a3f9232bac524dd0bac: Bootstrap starting.
I20260620 09:26:30.617861 23290 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13835b763e164a3f9232bac524dd0bac: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 09:26:30.618074 23290 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13835b763e164a3f9232bac524dd0bac: Bootstrap complete.
I20260620 09:26:30.618271 23290 raft_consensus.cc:348] T 00000000000000000000000000000000 P 13835b763e164a3f9232bac524dd0bac [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1d2095405b6a4278a42cc71c3c787e3e" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 40995 } } peers { permanent_uuid: "13835b763e164a3f9232bac524dd0bac" member_type: NON_VOTER last_known_addr { host: "127.16.43.189" port: 33891 } attrs { promote: true } }
I20260620 09:26:30.618362 23290 raft_consensus.cc:729] T 00000000000000000000000000000000 P 13835b763e164a3f9232bac524dd0bac [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 13835b763e164a3f9232bac524dd0bac, State: Initialized, Role: LEARNER
I20260620 09:26:30.618403 23290 consensus_queue.cc:260] T 00000000000000000000000000000000 P 13835b763e164a3f9232bac524dd0bac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1d2095405b6a4278a42cc71c3c787e3e" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 40995 } } peers { permanent_uuid: "13835b763e164a3f9232bac524dd0bac" member_type: NON_VOTER last_known_addr { host: "127.16.43.189" port: 33891 } attrs { promote: true } }
I20260620 09:26:30.618582 23290 sys_catalog.cc:564] T 00000000000000000000000000000000 P 13835b763e164a3f9232bac524dd0bac [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:26:30.618569 23293 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13835b763e164a3f9232bac524dd0bac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1d2095405b6a4278a42cc71c3c787e3e" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 40995 } } peers { permanent_uuid: "13835b763e164a3f9232bac524dd0bac" member_type: NON_VOTER last_known_addr { host: "127.16.43.189" port: 33891 } attrs { promote: true } } }
I20260620 09:26:30.618737 23293 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13835b763e164a3f9232bac524dd0bac [sys.catalog]: This master's current role is: LEARNER
I20260620 09:26:30.620309 23304 catalog_manager.cc:1259] Loaded cluster ID: 76574a0d1fa54743b9e218ccdfafe710
I20260620 09:26:30.620355 23304 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 13835b763e164a3f9232bac524dd0bac: loading cluster ID for follower catalog manager: success
I20260620 09:26:30.621503 23304 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 13835b763e164a3f9232bac524dd0bac: acquiring CA information for follower catalog manager: success
I20260620 09:26:30.621824 23304 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 13835b763e164a3f9232bac524dd0bac: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 09:26:30.660434 23305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:30.653635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42420 (local address 127.16.43.189:33891)
0620 09:26:30.653834 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:30.653835 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:30.653843 (+ 8us) server_negotiation.cc:407] Connection header received
0620 09:26:30.653866 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:30.653866 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:30.653876 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:30.653956 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:30.654615 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:30.655437 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:30.656191 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:30.656404 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:30.656489 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:30.657733 (+ 1244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:30.657734 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:30.657734 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:26:30.657742 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:30.659224 (+ 1482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:30.659628 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:30.659629 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:30.659630 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:30.659692 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:30.659901 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:30.659902 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:30.659903 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:30.660023 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:30.660129 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:30.660237 (+ 108us) server_negotiation.cc:299] Negotiation successful
0620 09:26:30.660323 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":90,"threads_started":1}
WARNING: no policy specified for oryx/127.16.43.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.43.188@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.43.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/krb5kdc/oryx_127.16.43.188.keytab.
Entry for principal oryx/127.16.43.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/krb5kdc/oryx_127.16.43.188.keytab.
I20260620 09:26:30.686939 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.43.188:46141
--webserver_interface=127.16.43.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/krb5kdc/oryx_127.16.43.188.keytab
--principal=oryx/127.16.43.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.43.148:34583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.43.190:40995,127.16.43.189:33891,127.16.43.188:46141
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:26:30.770859 23308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:26:30.771113 23308 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:26:30.771152 23308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:26:30.772390 23308 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:26:30.772466 23308 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:26:30.772490 23308 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:26:30.772521 23308 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:26:30.772547 23308 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:26:30.774137 23308 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:34583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.16.43.190:40995,127.16.43.189:33891,127.16.43.188:46141
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/krb5kdc/oryx_127.16.43.188.keytab
--principal=oryx/127.16.43.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.43.188:46141
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.16.43.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23308
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:26:30.774425 23308 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:26:30.774668 23308 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 09:26:30.779824 23308 server_base.cc:1034] running on GCE node
W20260620 09:26:30.779806 23313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:26:30.779809 23316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:26:30.779815 23314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:26:30.780247 23308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:26:30.780532 23308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:26:30.781771 23308 hybrid_clock.cc:648] HybridClock initialized: now 1781947590781738 us; error 37 us; skew 500 ppm
Jun 20 09:26:30 dist-test-slave-706q krb5kdc[22612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947590, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.43.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:26:30.784515 23308 init.cc:377] Logged in from keytab as oryx/127.16.43.188@KRBTEST.COM (short username oryx)
I20260620 09:26:30.785017 23308 webserver.cc:460] Webserver started at http://127.16.43.188:35817/ using document root <none> and password file <none>
I20260620 09:26:30.785212 23308 fs_manager.cc:359] Metadata directory not provided
I20260620 09:26:30.785301 23308 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:26:30.785413 23308 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:26:30.786666 23308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/master-2/data/instance:
uuid: "fc318f0d33f1422abfc6e301edf09951"
format_stamp: "Formatted at 2026-06-20 09:26:30 on dist-test-slave-706q"
server_key: "ac353e83dd8ed9e448bcebe304382918"
server_key_iv: "12f90541b9b43cd13e04b6a07370b750"
server_key_version: "encryptionkey@0"
I20260620 09:26:30.786993 23308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/master-2/wal/instance:
uuid: "fc318f0d33f1422abfc6e301edf09951"
format_stamp: "Formatted at 2026-06-20 09:26:30 on dist-test-slave-706q"
server_key: "ac353e83dd8ed9e448bcebe304382918"
server_key_iv: "12f90541b9b43cd13e04b6a07370b750"
server_key_version: "encryptionkey@0"
I20260620 09:26:30.788393 23308 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 09:26:30.789176 23323 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:30.789419 23308 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:26:30.789551 23308 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/master-2/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/master-2/wal
uuid: "fc318f0d33f1422abfc6e301edf09951"
format_stamp: "Formatted at 2026-06-20 09:26:30 on dist-test-slave-706q"
server_key: "ac353e83dd8ed9e448bcebe304382918"
server_key_iv: "12f90541b9b43cd13e04b6a07370b750"
server_key_version: "encryptionkey@0"
I20260620 09:26:30.789646 23308 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:26:30.804463 23308 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:26:30.806887 23308 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:26:30.807060 23308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:26:30.811251 23308 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.188:46141
I20260620 09:26:30.811287 23375 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.188:46141 every 8 connection(s)
I20260620 09:26:30.811676 23308 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/master-2/data/info.pb
I20260620 09:26:30.812106 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 23308
I20260620 09:26:30.813191 23376 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:26:30.814965 23376 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.43.190" port: 40995 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 09:26:30 dist-test-slave-706q krb5kdc[22612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947589, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.43.188@KRBTEST.COM
Jun 20 09:26:30 dist-test-slave-706q krb5kdc[22612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947590, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.43.188@KRBTEST.COM for oryx/127.16.43.190@KRBTEST.COM
I20260620 09:26:30.823312 23378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:30.812896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36120 (local address 127.16.43.188:46141)
0620 09:26:30.813111 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:30.813119 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:30.813164 (+ 45us) server_negotiation.cc:407] Connection header received
0620 09:26:30.813703 (+ 539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:30.813711 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:30.813764 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:30.813925 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:30.814550 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:30.815488 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:30.816487 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:30.817081 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:30.817191 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:30.819732 (+ 2541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:30.819739 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:30.819739 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:26:30.819746 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:30.821575 (+ 1829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:30.822048 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:30.822051 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:30.822051 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:30.822133 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:30.822421 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:30.822422 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:30.822422 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:30.822699 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:30.822804 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:30.822988 (+ 184us) server_negotiation.cc:299] Negotiation successful
0620 09:26:30.823080 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":166,"thread_start_us":99,"threads_started":1}
I20260620 09:26:30.824610 22697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:30.815871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40556 (local address 127.16.43.190:40995)
0620 09:26:30.815922 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:30.815924 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:30.816074 (+ 150us) server_negotiation.cc:407] Connection header received
0620 09:26:30.816136 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:30.816137 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:30.816147 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:30.816208 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:30.816675 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:30.817376 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:30.818325 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:30.818553 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:30.818672 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:30.821361 (+ 2689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:30.821363 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:30.821364 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:26:30.821373 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:30.823184 (+ 1811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:30.823692 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:30.823694 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:30.823694 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:30.823758 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:30.823984 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:30.823985 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:30.823986 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:30.824167 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:30.824251 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:30.824458 (+ 207us) server_negotiation.cc:299] Negotiation successful
0620 09:26:30.824537 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 09:26:30.824640 23379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:30.815723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:40995 (local address 127.0.0.1:40556)
0620 09:26:30.815967 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:30.816000 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:30.816069 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:30.816232 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:30.816234 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:30.816496 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:30.816593 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:30.816595 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:30.817488 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:30.817489 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:30.818190 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:30.818193 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:30.818711 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:30.818712 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:30.818824 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:30.819435 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:26:30.819444 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:26:30.821252 (+ 1808us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:26:30.823273 (+ 2021us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:30.823280 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:30.823282 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:30.823610 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:30.823849 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:30.823850 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:30.823851 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:30.823920 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:30.824237 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:26:30.824243 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:26:30.824301 (+ 58us) client_negotiation.cc:769] Sending connection context
0620 09:26:30.824403 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 09:26:30.824515 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"mutex_wait_us":1,"thread_start_us":96,"threads_started":1}
I20260620 09:26:30.825119 23376 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.43.189" port: 33891 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 09:26:30 dist-test-slave-706q krb5kdc[22612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947590, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.43.188@KRBTEST.COM for oryx/127.16.43.189@KRBTEST.COM
I20260620 09:26:30.832410 23379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:30.825447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.189:33891 (local address 127.0.0.1:42432)
0620 09:26:30.825497 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:30.825509 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:30.825565 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:30.825809 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:30.825810 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:30.825914 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:30.825955 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:30.825957 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:30.826800 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:30.826801 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:30.827346 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:30.827348 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:30.828078 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:30.828079 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:30.828136 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:30.828628 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:26:30.828639 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:26:30.829860 (+ 1221us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:26:30.831343 (+ 1483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:30.831344 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:30.831344 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:30.831507 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:30.831853 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:30.831853 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:30.831854 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:30.831908 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:30.832177 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:26:30.832178 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:26:30.832215 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 09:26:30.832262 (+ 47us) client_negotiation.cc:240] Negotiation successful
0620 09:26:30.832344 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17}
I20260620 09:26:30.832450 23305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:30.825541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42432 (local address 127.16.43.189:33891)
0620 09:26:30.825639 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:30.825642 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:30.825653 (+ 11us) server_negotiation.cc:407] Connection header received
0620 09:26:30.825681 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:30.825682 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:30.825698 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:30.825796 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:30.826049 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:30.826679 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:30.827468 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:30.827996 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:30.828060 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:30.829963 (+ 1903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:30.829965 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:30.829966 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:26:30.829973 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:30.831256 (+ 1283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:30.831715 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:30.831716 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:30.831717 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:30.831783 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:30.831972 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:30.831973 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:30.831973 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:30.832115 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:30.832179 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:30.832302 (+ 123us) server_negotiation.cc:299] Negotiation successful
0620 09:26:30.832382 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 09:26:30.832991 23376 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.43.188" port: 46141 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 09:26:30 dist-test-slave-706q krb5kdc[22612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947590, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.43.188@KRBTEST.COM for oryx/127.16.43.188@KRBTEST.COM
I20260620 09:26:30.839144 23379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:30.833305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.188:46141 (local address 127.0.0.1:36132)
0620 09:26:30.833362 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:30.833374 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:30.833418 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:30.833515 (+ 97us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:30.833515 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:30.833644 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:30.833675 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:30.833676 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:30.834274 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:30.834275 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:30.834820 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:30.834822 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:30.835147 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:30.835148 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:30.835191 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:30.835716 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:26:30.835721 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:26:30.836878 (+ 1157us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:26:30.838209 (+ 1331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:30.838210 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:30.838210 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:30.838364 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:30.838564 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:30.838564 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:30.838565 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:30.838617 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:30.838872 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:26:30.838873 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:26:30.838907 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 09:26:30.838958 (+ 51us) client_negotiation.cc:240] Negotiation successful
0620 09:26:30.839035 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260620 09:26:30.839156 23378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:30.833363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36132 (local address 127.16.43.188:46141)
0620 09:26:30.833420 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:30.833422 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:30.833429 (+ 7us) server_negotiation.cc:407] Connection header received
0620 09:26:30.833462 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:30.833462 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:30.833470 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:30.833507 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:30.833739 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:30.834196 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:30.834897 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:30.835076 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:30.835122 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:30.836954 (+ 1832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:30.836956 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:30.836956 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:26:30.836963 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:30.838133 (+ 1170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:30.838429 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:30.838430 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:30.838430 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:30.838476 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:30.838673 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:30.838673 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:30.838674 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:30.838815 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:30.838866 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:30.838973 (+ 107us) server_negotiation.cc:299] Negotiation successful
0620 09:26:30.839079 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":47,"server-negotiator.queue_time_us":40}
I20260620 09:26:30.840865 23376 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc318f0d33f1422abfc6e301edf09951: Bootstrap starting.
I20260620 09:26:30.841696 23376 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fc318f0d33f1422abfc6e301edf09951: Neither blocks nor log segments found. Creating new log.
I20260620 09:26:30.841979 23376 log.cc:824] T 00000000000000000000000000000000 P fc318f0d33f1422abfc6e301edf09951: Log is configured to *not* fsync() on all Append() calls
I20260620 09:26:30.842773 23376 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc318f0d33f1422abfc6e301edf09951: No bootstrap required, opened a new log
I20260620 09:26:30.844323 23376 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc318f0d33f1422abfc6e301edf09951 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d2095405b6a4278a42cc71c3c787e3e" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 40995 } } peers { permanent_uuid: "13835b763e164a3f9232bac524dd0bac" member_type: VOTER last_known_addr { host: "127.16.43.189" port: 33891 } } peers { permanent_uuid: "fc318f0d33f1422abfc6e301edf09951" member_type: VOTER last_known_addr { host: "127.16.43.188" port: 46141 } }
I20260620 09:26:30.844475 23376 raft_consensus.cc:374] T 00000000000000000000000000000000 P fc318f0d33f1422abfc6e301edf09951 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:26:30.844522 23376 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc318f0d33f1422abfc6e301edf09951 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc318f0d33f1422abfc6e301edf09951, State: Initialized, Role: FOLLOWER
I20260620 09:26:30.844671 23376 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc318f0d33f1422abfc6e301edf09951 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d2095405b6a4278a42cc71c3c787e3e" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 40995 } } peers { permanent_uuid: "13835b763e164a3f9232bac524dd0bac" member_type: VOTER last_known_addr { host: "127.16.43.189" port: 33891 } } peers { permanent_uuid: "fc318f0d33f1422abfc6e301edf09951" member_type: VOTER last_known_addr { host: "127.16.43.188" port: 46141 } }
I20260620 09:26:30.844980 23376 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc318f0d33f1422abfc6e301edf09951 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:26:30.844942 23381 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc318f0d33f1422abfc6e301edf09951 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d2095405b6a4278a42cc71c3c787e3e" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 40995 } } peers { permanent_uuid: "13835b763e164a3f9232bac524dd0bac" member_type: VOTER last_known_addr { host: "127.16.43.189" port: 33891 } } peers { permanent_uuid: "fc318f0d33f1422abfc6e301edf09951" member_type: VOTER last_known_addr { host: "127.16.43.188" port: 46141 } } }
I20260620 09:26:30.845098 23381 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc318f0d33f1422abfc6e301edf09951 [sys.catalog]: This master's current role is: FOLLOWER
W20260620 09:26:30.846839 23392 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fc318f0d33f1422abfc6e301edf09951: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 09:26:30.846902 23392 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 09:26:30.849064 23308 master_runner.cc:428] Detected that this master fc318f0d33f1422abfc6e301edf09951 is joining an existing cluster
I20260620 09:26:30.849138 23308 master_runner.cc:432] Initiating AddMaster RPC to add 127.16.43.188:46141
I20260620 09:26:30.857159 23378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:30.850683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36142 (local address 127.16.43.188:46141)
0620 09:26:30.850714 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:30.850716 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:30.850802 (+ 86us) server_negotiation.cc:407] Connection header received
0620 09:26:30.850880 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:30.850880 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:30.850888 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:30.850913 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:30.851305 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:30.852036 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:30.853112 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:30.853388 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:30.853425 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:30.854571 (+ 1146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:30.854573 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:30.854573 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:26:30.854580 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:30.856017 (+ 1437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:30.856378 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:30.856379 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:30.856380 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:30.856443 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:30.856642 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:30.856643 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:30.856644 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:30.856755 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:30.856830 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:30.856963 (+ 133us) server_negotiation.cc:299] Negotiation successful
0620 09:26:30.857080 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":57,"server-negotiator.queue_time_us":17}
I20260620 09:26:30.857170 23400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:30.850562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.188:46141 (local address 127.0.0.1:36142)
0620 09:26:30.850755 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:30.850763 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:30.850820 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:30.850963 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:30.850964 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:30.851181 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:30.851234 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:30.851236 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:30.852133 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:30.852134 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:30.853024 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:30.853026 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:30.853504 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:30.853505 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:30.853556 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:30.854044 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:26:30.854048 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:26:30.854467 (+ 419us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:26:30.856112 (+ 1645us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:30.856113 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:30.856114 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:30.856291 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:30.856518 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:30.856519 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:30.856519 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:30.856572 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:30.856820 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:26:30.856820 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:26:30.856859 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 09:26:30.856925 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 09:26:30.857031 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"mutex_wait_us":56,"thread_start_us":55,"threads_started":1}
I20260620 09:26:30.875063 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 22719
I20260620 09:26:30.880355 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.43.129:37091
--local_ip_for_outbound_sockets=127.16.43.129
--tserver_master_addrs=127.16.43.190:40995
--webserver_port=42355
--webserver_interface=127.16.43.129
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/krb5kdc/oryx_127.16.43.129.keytab
--principal=oryx/127.16.43.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.43.148:34583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.16.43.190:40995,127.16.43.189:33891
--tserver_master_addrs=127.16.43.190:40995,127.16.43.189:33891,127.16.43.188:46141 with env {KRB5CCNAME=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 09:26:30.884277 22697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:30.850202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40566 (local address 127.16.43.190:40995)
0620 09:26:30.850251 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:30.850253 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:30.850407 (+ 154us) server_negotiation.cc:407] Connection header received
0620 09:26:30.850516 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:30.850517 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:30.850529 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:30.850559 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:30.851007 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:30.851834 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:30.853034 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:30.853345 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:30.853413 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:30.880948 (+ 27535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:30.880951 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:30.880952 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:26:30.880961 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:30.882811 (+ 1850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:30.883309 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:30.883311 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:30.883311 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:30.883369 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:30.883612 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:30.883613 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:30.883614 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:30.883769 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:30.883813 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:30.884033 (+ 220us) server_negotiation.cc:299] Negotiation successful
0620 09:26:30.884128 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 09:26:30.884706 23305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:30.850448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42448 (local address 127.16.43.189:33891)
0620 09:26:30.850491 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:30.850494 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:30.850600 (+ 106us) server_negotiation.cc:407] Connection header received
0620 09:26:30.850718 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:30.850720 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:30.850731 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:30.850766 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:30.851248 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:30.852040 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:30.853171 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:30.853481 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:30.853520 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:30.881549 (+ 28029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:30.881551 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:30.881551 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:26:30.881560 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:30.883435 (+ 1875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:30.883865 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:30.883867 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:30.883867 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:30.883924 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:30.884115 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:30.884116 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:30.884116 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:30.884255 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:30.884297 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:30.884491 (+ 194us) server_negotiation.cc:299] Negotiation successful
0620 09:26:30.884611 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 09:26:30.885018 23399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:30.850402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.189:33891 (local address 127.0.0.1:42448)
0620 09:26:30.850540 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:30.850551 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:30.850625 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:30.850835 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:30.850836 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:30.851095 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:30.851150 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:30.851153 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:30.852128 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:30.852129 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:30.853080 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:30.853081 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:30.853589 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:30.853590 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:30.853645 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:30.880245 (+ 26600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:26:30.880252 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:26:30.881176 (+ 924us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:26:30.883570 (+ 2394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:30.883571 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:30.883572 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:30.883776 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:30.883996 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:30.883996 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:30.883997 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:30.884049 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:30.884343 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:26:30.884344 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:26:30.884384 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 09:26:30.884792 (+ 408us) client_negotiation.cc:240] Negotiation successful
0620 09:26:30.884905 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"mutex_wait_us":1,"thread_start_us":61,"threads_started":1}
I20260620 09:26:30.885555 23398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:30.850174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:40995 (local address 127.0.0.1:40566)
0620 09:26:30.850330 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:30.850343 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:30.850439 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:30.850622 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:30.850622 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:30.850834 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:30.850916 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:30.850918 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:30.851961 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:30.851962 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:30.852921 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:30.852923 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:30.853522 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:30.853523 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:30.853623 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:30.880361 (+ 26738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:26:30.880366 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:26:30.880820 (+ 454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:26:30.882947 (+ 2127us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:30.882948 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:30.882948 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:30.883220 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:30.883438 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:30.883439 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:30.883439 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:30.883498 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:30.883859 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:26:30.883860 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:26:30.883921 (+ 61us) client_negotiation.cc:769] Sending connection context
0620 09:26:30.884931 (+ 1010us) client_negotiation.cc:240] Negotiation successful
0620 09:26:30.885344 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":79,"threads_started":1}
Jun 20 09:26:30 dist-test-slave-706q krb5kdc[22612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947589, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.43.190@KRBTEST.COM for oryx/127.16.43.188@KRBTEST.COM
I20260620 09:26:30.895996 23214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:30.887913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.188:46141 (local address 127.0.0.1:36158)
0620 09:26:30.887963 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:30.887973 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:30.888008 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:30.888145 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:30.888146 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:30.888316 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:30.888364 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:30.888366 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:30.889363 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:30.889365 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:30.890230 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:30.890233 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:30.890733 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:30.890734 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:30.890802 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:30.891269 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:26:30.891274 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:26:30.892891 (+ 1617us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:26:30.894695 (+ 1804us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:30.894696 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:30.894697 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:30.894910 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:30.895112 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:30.895112 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:30.895113 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:30.895181 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:30.895450 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:26:30.895451 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:26:30.895495 (+ 44us) client_negotiation.cc:769] Sending connection context
0620 09:26:30.895783 (+ 288us) client_negotiation.cc:240] Negotiation successful
0620 09:26:30.895891 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260620 09:26:30.896014 23378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:30.887975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36158 (local address 127.16.43.188:46141)
0620 09:26:30.888014 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:30.888016 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:30.888026 (+ 10us) server_negotiation.cc:407] Connection header received
0620 09:26:30.888062 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:30.888063 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:30.888077 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:30.888126 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:30.888461 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:30.889267 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:30.890314 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:30.890648 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:30.890727 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:30.892968 (+ 2241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:30.892970 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:30.892970 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:26:30.892979 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:30.894599 (+ 1620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:30.894983 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:30.894984 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:30.894985 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:30.895042 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:30.895247 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:30.895248 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:30.895248 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:30.895382 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:30.895449 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:30.895836 (+ 387us) server_negotiation.cc:299] Negotiation successful
0620 09:26:30.895918 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 09:26:30.896924 22657 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.16.43.188:46141: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "fc318f0d33f1422abfc6e301edf09951"
member_type: NON_VOTER
last_known_addr {
host: "127.16.43.188"
port: 46141
}
attrs {
promote: true
}
}
dest_uuid: "1d2095405b6a4278a42cc71c3c787e3e"
cas_config_opid_index: 5
I20260620 09:26:30.897186 22657 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1d2095405b6a4278a42cc71c3c787e3e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1d2095405b6a4278a42cc71c3c787e3e" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 40995 } } peers { permanent_uuid: "13835b763e164a3f9232bac524dd0bac" member_type: NON_VOTER last_known_addr { host: "127.16.43.189" port: 33891 } attrs { promote: true } } peers { permanent_uuid: "fc318f0d33f1422abfc6e301edf09951" member_type: NON_VOTER last_known_addr { host: "127.16.43.188" port: 46141 } attrs { promote: true } }
I20260620 09:26:30.897696 23215 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1d2095405b6a4278a42cc71c3c787e3e [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER fc318f0d33f1422abfc6e301edf09951 (127.16.43.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1d2095405b6a4278a42cc71c3c787e3e" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 40995 } } peers { permanent_uuid: "13835b763e164a3f9232bac524dd0bac" member_type: NON_VOTER last_known_addr { host: "127.16.43.189" port: 33891 } attrs { promote: true } } peers { permanent_uuid: "fc318f0d33f1422abfc6e301edf09951" member_type: NON_VOTER last_known_addr { host: "127.16.43.188" port: 46141 } attrs { promote: true } } }
I20260620 09:26:30.898706 23215 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.16.43.188:46141
I20260620 09:26:30.898775 23402 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d2095405b6a4278a42cc71c3c787e3e [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "1d2095405b6a4278a42cc71c3c787e3e" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1d2095405b6a4278a42cc71c3c787e3e" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 40995 } } peers { permanent_uuid: "13835b763e164a3f9232bac524dd0bac" member_type: NON_VOTER last_known_addr { host: "127.16.43.189" port: 33891 } attrs { promote: true } } peers { permanent_uuid: "fc318f0d33f1422abfc6e301edf09951" member_type: NON_VOTER last_known_addr { host: "127.16.43.188" port: 46141 } attrs { promote: true } } }
I20260620 09:26:30.898862 23402 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d2095405b6a4278a42cc71c3c787e3e [sys.catalog]: This master's current role is: LEADER
I20260620 09:26:30.899261 23308 master.cc:537] Master@127.16.43.188:46141 shutting down...
I20260620 09:26:30.899410 23350 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc318f0d33f1422abfc6e301edf09951 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:26:30.901029 23350 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fc318f0d33f1422abfc6e301edf09951 [term 1 FOLLOWER]: Refusing update from remote peer 1d2095405b6a4278a42cc71c3c787e3e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 09:26:30.902209 23402 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1d2095405b6a4278a42cc71c3c787e3e [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc318f0d33f1422abfc6e301edf09951" member_type: NON_VOTER last_known_addr { host: "127.16.43.188" port: 46141 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 09:26:30.902940 22643 consensus_peers.cc:487] T 00000000000000000000000000000000 P 1d2095405b6a4278a42cc71c3c787e3e -> Peer fc318f0d33f1422abfc6e301edf09951 (127.16.43.188:46141): Couldn't send request to peer fc318f0d33f1422abfc6e301edf09951. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260620 09:26:30.905282 23214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:30.898925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.189:33891 (local address 127.0.0.1:42450)
0620 09:26:30.898959 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:30.898971 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:30.899016 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:30.901844 (+ 2828us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:30.901846 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:30.901848 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:26:30.901910 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:30.901912 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:30.902978 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:30.902979 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:30.904252 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:30.904255 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:30.904987 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:30.904988 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:30.905058 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:30.905084 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 09:26:30.905142 (+ 58us) client_negotiation.cc:240] Negotiation successful
0620 09:26:30.905215 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19}
I20260620 09:26:30.905714 23305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:30.899022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42450 (local address 127.16.43.189:33891)
0620 09:26:30.901633 (+ 2611us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:30.901635 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:30.901646 (+ 11us) server_negotiation.cc:407] Connection header received
0620 09:26:30.901672 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:30.901674 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:30.901698 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:30.901774 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:26:30.902006 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:30.902781 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:30.904352 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:30.904897 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:30.904978 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:30.905053 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:30.905580 (+ 527us) server_negotiation.cc:299] Negotiation successful
0620 09:26:30.905630 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2582}
I20260620 09:26:30.905901 23264 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 13835b763e164a3f9232bac524dd0bac [term 1 LEARNER]: Refusing update from remote peer 1d2095405b6a4278a42cc71c3c787e3e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 09:26:30.906071 23402 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1d2095405b6a4278a42cc71c3c787e3e [LEADER]: Connected to new peer: Peer: permanent_uuid: "13835b763e164a3f9232bac524dd0bac" member_type: NON_VOTER last_known_addr { host: "127.16.43.189" port: 33891 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 09:26:30.906703 23308 raft_consensus.cc:2229] T 00000000000000000000000000000000 P fc318f0d33f1422abfc6e301edf09951 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:26:30.906790 23308 raft_consensus.cc:2258] T 00000000000000000000000000000000 P fc318f0d33f1422abfc6e301edf09951 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:26:30.906827 23308 tablet_replica.cc:330] T 00000000000000000000000000000000 P fc318f0d33f1422abfc6e301edf09951: stopping tablet replica
I20260620 09:26:30.906875 23264 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 13835b763e164a3f9232bac524dd0bac [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER fc318f0d33f1422abfc6e301edf09951 (127.16.43.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1d2095405b6a4278a42cc71c3c787e3e" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 40995 } } peers { permanent_uuid: "13835b763e164a3f9232bac524dd0bac" member_type: NON_VOTER last_known_addr { host: "127.16.43.189" port: 33891 } attrs { promote: true } } peers { permanent_uuid: "fc318f0d33f1422abfc6e301edf09951" member_type: NON_VOTER last_known_addr { host: "127.16.43.188" port: 46141 } attrs { promote: true } } }
I20260620 09:26:30.908296 23404 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13835b763e164a3f9232bac524dd0bac [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1d2095405b6a4278a42cc71c3c787e3e" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1d2095405b6a4278a42cc71c3c787e3e" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 40995 } } peers { permanent_uuid: "13835b763e164a3f9232bac524dd0bac" member_type: NON_VOTER last_known_addr { host: "127.16.43.189" port: 33891 } attrs { promote: true } } peers { permanent_uuid: "fc318f0d33f1422abfc6e301edf09951" member_type: NON_VOTER last_known_addr { host: "127.16.43.188" port: 46141 } attrs { promote: true } } }
I20260620 09:26:30.908385 23404 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13835b763e164a3f9232bac524dd0bac [sys.catalog]: This master's current role is: LEARNER
I20260620 09:26:30.908502 23293 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13835b763e164a3f9232bac524dd0bac [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d2095405b6a4278a42cc71c3c787e3e. Latest consensus state: current_term: 1 leader_uuid: "1d2095405b6a4278a42cc71c3c787e3e" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1d2095405b6a4278a42cc71c3c787e3e" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 40995 } } peers { permanent_uuid: "13835b763e164a3f9232bac524dd0bac" member_type: NON_VOTER last_known_addr { host: "127.16.43.189" port: 33891 } attrs { promote: true } } peers { permanent_uuid: "fc318f0d33f1422abfc6e301edf09951" member_type: NON_VOTER last_known_addr { host: "127.16.43.188" port: 46141 } attrs { promote: true } } }
I20260620 09:26:30.908586 23293 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13835b763e164a3f9232bac524dd0bac [sys.catalog]: This master's current role is: LEARNER
I20260620 09:26:30.909969 23402 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 1d2095405b6a4278a42cc71c3c787e3e: attempting to promote NON_VOTER 13835b763e164a3f9232bac524dd0bac to VOTER
I20260620 09:26:30.910195 23402 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1d2095405b6a4278a42cc71c3c787e3e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1d2095405b6a4278a42cc71c3c787e3e" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 40995 } } peers { permanent_uuid: "13835b763e164a3f9232bac524dd0bac" member_type: VOTER last_known_addr { host: "127.16.43.189" port: 33891 } attrs { promote: false } } peers { permanent_uuid: "fc318f0d33f1422abfc6e301edf09951" member_type: NON_VOTER last_known_addr { host: "127.16.43.188" port: 46141 } attrs { promote: true } }
I20260620 09:26:30.910987 23214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:30.910852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.188:46141 (local address 127.0.0.1:36172)
0620 09:26:30.910916 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:30.910957 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.188:46141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260620 09:26:30.911022 23264 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 13835b763e164a3f9232bac524dd0bac [term 1 LEARNER]: Refusing update from remote peer 1d2095405b6a4278a42cc71c3c787e3e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260620 09:26:30.911410 23214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:30.911335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.188:46141 (local address 127.0.0.1:36182)
0620 09:26:30.911367 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:30.911388 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.188:46141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20260620 09:26:30.911564 22643 consensus_peers.cc:487] T 00000000000000000000000000000000 P 1d2095405b6a4278a42cc71c3c787e3e -> Peer fc318f0d33f1422abfc6e301edf09951 (127.16.43.188:46141): Couldn't send request to peer fc318f0d33f1422abfc6e301edf09951. Status: Network error: Client connection negotiation failed: client connection to 127.16.43.188:46141: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 09:26:30.911614 23402 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1d2095405b6a4278a42cc71c3c787e3e [LEADER]: Connected to new peer: Peer: permanent_uuid: "13835b763e164a3f9232bac524dd0bac" member_type: VOTER last_known_addr { host: "127.16.43.189" port: 33891 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260620 09:26:30.912281 23215 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1d2095405b6a4278a42cc71c3c787e3e [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 13835b763e164a3f9232bac524dd0bac (127.16.43.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1d2095405b6a4278a42cc71c3c787e3e" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 40995 } } peers { permanent_uuid: "13835b763e164a3f9232bac524dd0bac" member_type: VOTER last_known_addr { host: "127.16.43.189" port: 33891 } attrs { promote: false } } peers { permanent_uuid: "fc318f0d33f1422abfc6e301edf09951" member_type: NON_VOTER last_known_addr { host: "127.16.43.188" port: 46141 } attrs { promote: true } } }
I20260620 09:26:30.912444 23264 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 13835b763e164a3f9232bac524dd0bac [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 13835b763e164a3f9232bac524dd0bac (127.16.43.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1d2095405b6a4278a42cc71c3c787e3e" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 40995 } } peers { permanent_uuid: "13835b763e164a3f9232bac524dd0bac" member_type: VOTER last_known_addr { host: "127.16.43.189" port: 33891 } attrs { promote: false } } peers { permanent_uuid: "fc318f0d33f1422abfc6e301edf09951" member_type: NON_VOTER last_known_addr { host: "127.16.43.188" port: 46141 } attrs { promote: true } } }
I20260620 09:26:30.913192 23215 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d2095405b6a4278a42cc71c3c787e3e [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "1d2095405b6a4278a42cc71c3c787e3e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1d2095405b6a4278a42cc71c3c787e3e" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 40995 } } peers { permanent_uuid: "13835b763e164a3f9232bac524dd0bac" member_type: VOTER last_known_addr { host: "127.16.43.189" port: 33891 } attrs { promote: false } } peers { permanent_uuid: "fc318f0d33f1422abfc6e301edf09951" member_type: NON_VOTER last_known_addr { host: "127.16.43.188" port: 46141 } attrs { promote: true } } }
I20260620 09:26:30.913278 23215 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d2095405b6a4278a42cc71c3c787e3e [sys.catalog]: This master's current role is: LEADER
I20260620 09:26:30.913327 23215 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d2095405b6a4278a42cc71c3c787e3e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1d2095405b6a4278a42cc71c3c787e3e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1d2095405b6a4278a42cc71c3c787e3e" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 40995 } } peers { permanent_uuid: "13835b763e164a3f9232bac524dd0bac" member_type: VOTER last_known_addr { host: "127.16.43.189" port: 33891 } attrs { promote: false } } peers { permanent_uuid: "fc318f0d33f1422abfc6e301edf09951" member_type: NON_VOTER last_known_addr { host: "127.16.43.188" port: 46141 } attrs { promote: true } } }
I20260620 09:26:30.913367 23215 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d2095405b6a4278a42cc71c3c787e3e [sys.catalog]: This master's current role is: LEADER
I20260620 09:26:30.913421 23293 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13835b763e164a3f9232bac524dd0bac [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1d2095405b6a4278a42cc71c3c787e3e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1d2095405b6a4278a42cc71c3c787e3e" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 40995 } } peers { permanent_uuid: "13835b763e164a3f9232bac524dd0bac" member_type: VOTER last_known_addr { host: "127.16.43.189" port: 33891 } attrs { promote: false } } peers { permanent_uuid: "fc318f0d33f1422abfc6e301edf09951" member_type: NON_VOTER last_known_addr { host: "127.16.43.188" port: 46141 } attrs { promote: true } } }
I20260620 09:26:30.913496 23293 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13835b763e164a3f9232bac524dd0bac [sys.catalog]: This master's current role is: FOLLOWER
I20260620 09:26:30.920785 23308 master.cc:552] Master@127.16.43.188:46141 shutdown complete.
I20260620 09:26:30.920854 23308 master_runner.cc:305] Clearing existing system tablet
I20260620 09:26:30.920992 23308 fs_manager.cc:359] Metadata directory not provided
I20260620 09:26:30.921041 23308 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:26:30.921836 23308 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:26:30.922410 23406 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:30.922559 23308 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:26:30.922618 23308 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:30.922644 23308 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/master-2/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/master-2/wal
uuid: "fc318f0d33f1422abfc6e301edf09951"
format_stamp: "Formatted at 2026-06-20 09:26:30 on dist-test-slave-706q"
server_key: "ac353e83dd8ed9e448bcebe304382918"
server_key_iv: "12f90541b9b43cd13e04b6a07370b750"
server_key_version: "encryptionkey@0"
I20260620 09:26:30.923405 23308 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P fc318f0d33f1422abfc6e301edf09951: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:26:30.925051 23308 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P fc318f0d33f1422abfc6e301edf09951: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 09:26:30.925124 23308 log.cc:1196] T 00000000000000000000000000000000 P fc318f0d33f1422abfc6e301edf09951: Deleting WAL directory at /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260620 09:26:30.925365 23308 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P fc318f0d33f1422abfc6e301edf09951: Deleting consensus metadata
I20260620 09:26:30.925501 23308 master_runner.cc:315] Copying system tablet from 127.16.43.190:40995
I20260620 09:26:30.926146 23308 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P fc318f0d33f1422abfc6e301edf09951: tablet copy: Beginning tablet copy session from remote peer at address 127.16.43.190:40995
I20260620 09:26:30.933780 23412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:30.926459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:40995 (local address 127.0.0.1:40572)
0620 09:26:30.926592 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:30.926603 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:30.926662 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:30.926825 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:30.926826 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:30.927030 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:30.927082 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:30.927084 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:30.928151 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:30.928152 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:30.929103 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:30.929106 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:30.929624 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:30.929625 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:30.929683 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:30.930249 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:26:30.930255 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:26:30.930657 (+ 402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:26:30.932613 (+ 1956us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:30.932615 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:30.932615 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:30.932846 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:30.933076 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:30.933077 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:30.933078 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:30.933142 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:30.933433 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:26:30.933434 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:26:30.933486 (+ 52us) client_negotiation.cc:769] Sending connection context
0620 09:26:30.933560 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 09:26:30.933672 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":72,"threads_started":1}
I20260620 09:26:30.933822 22697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:30.926547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40572 (local address 127.16.43.190:40995)
0620 09:26:30.926597 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:30.926600 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:30.926652 (+ 52us) server_negotiation.cc:407] Connection header received
0620 09:26:30.926719 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:30.926720 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:30.926732 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:30.926816 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:30.927192 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:30.928046 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:30.929236 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:30.929530 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:30.929611 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:30.930756 (+ 1145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:30.930758 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:30.930758 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:26:30.930767 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:30.932514 (+ 1747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:30.932933 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:30.932934 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:30.932935 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:30.933001 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:30.933220 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:30.933221 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:30.933221 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:30.933361 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:30.933434 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:30.933610 (+ 176us) server_negotiation.cc:299] Negotiation successful
0620 09:26:30.933693 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 09:26:30.934500 22677 tablet_copy_service.cc:140] P 1d2095405b6a4278a42cc71c3c787e3e: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer fc318f0d33f1422abfc6e301edf09951 ({username='oryx', principal='oryx/127.16.43.188@KRBTEST.COM'} at 127.0.0.1:40572)
I20260620 09:26:30.934569 22677 tablet_copy_service.cc:161] P 1d2095405b6a4278a42cc71c3c787e3e: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer fc318f0d33f1422abfc6e301edf09951 at {username='oryx', principal='oryx/127.16.43.188@KRBTEST.COM'} at 127.0.0.1:40572: session id = fc318f0d33f1422abfc6e301edf09951-00000000000000000000000000000000
I20260620 09:26:30.935209 22677 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 1d2095405b6a4278a42cc71c3c787e3e: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 09:26:30.935732 23308 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 09:26:30.937206 23308 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P fc318f0d33f1422abfc6e301edf09951: tablet copy: Starting download of 0 data blocks...
I20260620 09:26:30.937371 23308 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P fc318f0d33f1422abfc6e301edf09951: tablet copy: Starting download of 1 WAL segments...
I20260620 09:26:30.938174 23308 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P fc318f0d33f1422abfc6e301edf09951: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 09:26:30.939114 22677 tablet_copy_service.cc:342] P 1d2095405b6a4278a42cc71c3c787e3e: Request end of tablet copy session fc318f0d33f1422abfc6e301edf09951-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.16.43.188@KRBTEST.COM'} at 127.0.0.1:40572
I20260620 09:26:30.939189 22677 tablet_copy_service.cc:433] P 1d2095405b6a4278a42cc71c3c787e3e: ending tablet copy session fc318f0d33f1422abfc6e301edf09951-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer fc318f0d33f1422abfc6e301edf09951
I20260620 09:26:30.942099 23308 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:26:30.943475 23413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:26:30.944661 23414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:26:30.944715 23416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:26:30.947721 23308 server_base.cc:1034] running on GCE node
I20260620 09:26:30.947899 23308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:26:30.948061 23308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:26:30.949208 23308 hybrid_clock.cc:648] HybridClock initialized: now 1781947590949188 us; error 44 us; skew 500 ppm
Jun 20 09:26:30 dist-test-slave-706q krb5kdc[22612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947590, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.43.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:26:30.950802 23308 init.cc:377] Logged in from keytab as oryx/127.16.43.188@KRBTEST.COM (short username oryx)
I20260620 09:26:30.951079 23308 webserver.cc:460] Webserver started at http://127.16.43.188:41105/ using document root <none> and password file <none>
I20260620 09:26:30.951208 23308 fs_manager.cc:359] Metadata directory not provided
I20260620 09:26:30.951242 23308 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:26:30.952036 23308 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:26:30.952666 23423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:30.952839 23308 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 09:26:30.952891 23308 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/master-2/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/master-2/wal
uuid: "fc318f0d33f1422abfc6e301edf09951"
format_stamp: "Formatted at 2026-06-20 09:26:30 on dist-test-slave-706q"
server_key: "ac353e83dd8ed9e448bcebe304382918"
server_key_iv: "12f90541b9b43cd13e04b6a07370b750"
server_key_version: "encryptionkey@0"
I20260620 09:26:30.952947 23308 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:26:30.972900 23308 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:26:30.974876 23308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:26:30.978492 23308 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.188:46141
I20260620 09:26:30.978538 23475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.188:46141 every 8 connection(s)
I20260620 09:26:30.978852 23308 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/master-2/data/info.pb
I20260620 09:26:30.979843 23476 sys_catalog.cc:263] Verifying existing consensus state
I20260620 09:26:30.980458 23476 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc318f0d33f1422abfc6e301edf09951: Bootstrap starting.
I20260620 09:26:30.983517 23476 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc318f0d33f1422abfc6e301edf09951: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 09:26:30.983817 23476 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc318f0d33f1422abfc6e301edf09951: Bootstrap complete.
I20260620 09:26:30.984047 23476 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc318f0d33f1422abfc6e301edf09951 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1d2095405b6a4278a42cc71c3c787e3e" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 40995 } } peers { permanent_uuid: "13835b763e164a3f9232bac524dd0bac" member_type: VOTER last_known_addr { host: "127.16.43.189" port: 33891 } attrs { promote: false } } peers { permanent_uuid: "fc318f0d33f1422abfc6e301edf09951" member_type: NON_VOTER last_known_addr { host: "127.16.43.188" port: 46141 } attrs { promote: true } }
I20260620 09:26:30.984144 23476 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc318f0d33f1422abfc6e301edf09951 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: fc318f0d33f1422abfc6e301edf09951, State: Initialized, Role: LEARNER
I20260620 09:26:30.984192 23476 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc318f0d33f1422abfc6e301edf09951 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1d2095405b6a4278a42cc71c3c787e3e" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 40995 } } peers { permanent_uuid: "13835b763e164a3f9232bac524dd0bac" member_type: VOTER last_known_addr { host: "127.16.43.189" port: 33891 } attrs { promote: false } } peers { permanent_uuid: "fc318f0d33f1422abfc6e301edf09951" member_type: NON_VOTER last_known_addr { host: "127.16.43.188" port: 46141 } attrs { promote: true } }
I20260620 09:26:30.984411 23476 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc318f0d33f1422abfc6e301edf09951 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:26:30.984381 23479 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc318f0d33f1422abfc6e301edf09951 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1d2095405b6a4278a42cc71c3c787e3e" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 40995 } } peers { permanent_uuid: "13835b763e164a3f9232bac524dd0bac" member_type: VOTER last_known_addr { host: "127.16.43.189" port: 33891 } attrs { promote: false } } peers { permanent_uuid: "fc318f0d33f1422abfc6e301edf09951" member_type: NON_VOTER last_known_addr { host: "127.16.43.188" port: 46141 } attrs { promote: true } } }
I20260620 09:26:30.984596 23479 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc318f0d33f1422abfc6e301edf09951 [sys.catalog]: This master's current role is: LEARNER
I20260620 09:26:30.985725 23490 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260620 09:26:30.991402 23401 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:26:30.991701 23401 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:26:30.991734 23401 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:26:30.992925 23401 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:26:30.992959 23401 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:26:30.992987 23401 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.43.129
I20260620 09:26:30.994536 23401 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:34583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/krb5kdc/oryx_127.16.43.129.keytab
--principal=oryx/127.16.43.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.43.129:37091
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.43.129
--webserver_port=42355
--tserver_master_addrs=127.16.43.190:40995,127.16.43.189:33891,127.16.43.188:46141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23401
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.43.129
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:26:30.994890 23401 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:26:30.995146 23401 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:26:31.001348 23495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:26:31.001385 23496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:26:31.001673 23498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:26:31.001786 23401 server_base.cc:1034] running on GCE node
I20260620 09:26:31.001992 23401 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:26:31.002278 23401 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:26:31.003445 23401 hybrid_clock.cc:648] HybridClock initialized: now 1781947591003422 us; error 38 us; skew 500 ppm
Jun 20 09:26:30 dist-test-slave-706q krb5kdc[22612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947590, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.43.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:26:31.006160 23401 init.cc:377] Logged in from keytab as oryx/127.16.43.129@KRBTEST.COM (short username oryx)
I20260620 09:26:31.006610 23401 webserver.cc:460] Webserver started at http://127.16.43.129:42355/ using document root <none> and password file <none>
I20260620 09:26:31.006886 23401 fs_manager.cc:359] Metadata directory not provided
I20260620 09:26:31.006946 23401 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:26:31.008560 23401 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:26:31.009380 23505 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:31.009665 23401 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:26:31.009789 23401 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/ts-0/wal
uuid: "625889083e23420fb414432cf88b0b51"
format_stamp: "Formatted at 2026-06-20 09:26:29 on dist-test-slave-706q"
server_key: "f0a471b8319018ffbbd50c4f0fc4e1c5"
server_key_iv: "73090e7bc29894b768ea31c99f082bd2"
server_key_version: "encryptionkey@0"
I20260620 09:26:31.010170 23401 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:26:31.032729 23401 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:26:31.035082 23401 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:26:31.035265 23401 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:26:31.035521 23401 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:26:31.035872 23401 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:26:31.035929 23401 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:31.035974 23401 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:26:31.036005 23401 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:31.041920 23401 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.129:37091
I20260620 09:26:31.041998 23618 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.129:37091 every 8 connection(s)
I20260620 09:26:31.042428 23401 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/ts-0/data/info.pb
I20260620 09:26:31.046537 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 23401
I20260620 09:26:31.046723 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 22853
Jun 20 09:26:31 dist-test-slave-706q krb5kdc[22612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947590, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.43.129@KRBTEST.COM for oryx/127.16.43.188@KRBTEST.COM
Jun 20 09:26:31 dist-test-slave-706q krb5kdc[22612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947590, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.43.129@KRBTEST.COM for oryx/127.16.43.189@KRBTEST.COM
I20260620 09:26:31.053444 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.43.130:38265
--local_ip_for_outbound_sockets=127.16.43.130
--tserver_master_addrs=127.16.43.190:40995
--webserver_port=33639
--webserver_interface=127.16.43.130
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/krb5kdc/oryx_127.16.43.130.keytab
--principal=oryx/127.16.43.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.43.148:34583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.16.43.190:40995,127.16.43.189:33891
--tserver_master_addrs=127.16.43.190:40995,127.16.43.189:33891,127.16.43.188:46141 with env {KRB5CCNAME=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 09:26:31.055348 23624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:31.043291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.188:46141 (local address 127.16.43.129:52325)
0620 09:26:31.043505 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:31.043586 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:31.044697 (+ 1111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:31.045009 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:31.045009 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:31.045456 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:31.045654 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:31.045660 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:31.047720 (+ 2060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:31.047721 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:31.048521 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:31.048523 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:31.048993 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:31.048994 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:31.049064 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:31.049694 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:26:31.049705 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:26:31.051391 (+ 1686us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:26:31.053489 (+ 2098us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:31.053497 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:31.053500 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:31.053894 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:31.054107 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:31.054108 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:31.054109 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:31.054257 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:31.054594 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:26:31.054601 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:26:31.054791 (+ 190us) client_negotiation.cc:769] Sending connection context
0620 09:26:31.054908 (+ 117us) client_negotiation.cc:240] Negotiation successful
0620 09:26:31.055086 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"mutex_wait_us":10,"thread_start_us":83,"threads_started":1}
Jun 20 09:26:31 dist-test-slave-706q krb5kdc[22612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947590, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.43.129@KRBTEST.COM for oryx/127.16.43.190@KRBTEST.COM
I20260620 09:26:31.055677 23305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:31.043872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.129:37801 (local address 127.16.43.189:33891)
0620 09:26:31.043927 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:31.043929 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:31.043999 (+ 70us) server_negotiation.cc:407] Connection header received
0620 09:26:31.044833 (+ 834us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:31.044835 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:31.044849 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:31.044899 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:31.045763 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:31.046339 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:31.048475 (+ 2136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:31.048773 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:31.048841 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:31.052189 (+ 3348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:31.052191 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:31.052191 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:26:31.052200 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:31.054448 (+ 2248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:31.054823 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:31.054824 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:31.054824 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:31.054880 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:31.055075 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:31.055076 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:31.055077 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:31.055225 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:31.055266 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:31.055473 (+ 207us) server_negotiation.cc:299] Negotiation successful
0620 09:26:31.055586 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 09:26:31.056072 23626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:31.043815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.189:33891 (local address 127.16.43.129:37801)
0620 09:26:31.043945 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:31.043954 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:31.044721 (+ 767us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:31.044989 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:31.044993 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:31.045442 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:31.045653 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:31.045658 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:31.047522 (+ 1864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:31.047523 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:31.048364 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:31.048366 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:31.048865 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:31.048866 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:31.049007 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:31.049739 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:26:31.049745 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:26:31.051789 (+ 2044us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:26:31.054535 (+ 2746us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:31.054535 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:31.054536 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:31.054737 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:31.054953 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:31.054953 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:31.054954 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:31.055004 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:31.055312 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:26:31.055313 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:26:31.055350 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 09:26:31.055805 (+ 455us) client_negotiation.cc:240] Negotiation successful
0620 09:26:31.055959 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":63,"threads_started":1}
I20260620 09:26:31.056576 23623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:31.043288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.129:52325 (local address 127.16.43.188:46141)
0620 09:26:31.043501 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:31.043504 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:31.044845 (+ 1341us) server_negotiation.cc:407] Connection header received
0620 09:26:31.044871 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:31.044872 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:31.044885 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:31.044946 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:31.045767 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:31.046762 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:31.048604 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:31.048907 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:31.048994 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:31.051505 (+ 2511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:31.051507 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:31.051507 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:26:31.051515 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:31.053397 (+ 1882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:31.053983 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:31.053985 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:31.053985 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:31.054049 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:31.054348 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:31.054349 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:31.054350 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:31.054505 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:31.054569 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:31.056423 (+ 1854us) server_negotiation.cc:299] Negotiation successful
0620 09:26:31.056512 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":75,"threads_started":1}
I20260620 09:26:31.057113 23619 heartbeater.cc:347] Connected to a master server at 127.16.43.188:46141
I20260620 09:26:31.057221 23619 heartbeater.cc:464] Registering TS with master...
I20260620 09:26:31.057294 23621 heartbeater.cc:347] Connected to a master server at 127.16.43.189:33891
I20260620 09:26:31.057335 23621 heartbeater.cc:464] Registering TS with master...
I20260620 09:26:31.057459 23619 heartbeater.cc:511] Master 127.16.43.188:46141 requested a full tablet report, sending...
I20260620 09:26:31.057449 23621 heartbeater.cc:511] Master 127.16.43.189:33891 requested a full tablet report, sending...
I20260620 09:26:31.058089 23440 ts_manager.cc:194] Registered new tserver with Master: 625889083e23420fb414432cf88b0b51 (127.16.43.129:37091)
I20260620 09:26:31.058087 23254 ts_manager.cc:194] Registered new tserver with Master: 625889083e23420fb414432cf88b0b51 (127.16.43.129:37091)
I20260620 09:26:31.059641 22697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:31.043693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.129:53467 (local address 127.16.43.190:40995)
0620 09:26:31.043756 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:31.043758 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:31.043880 (+ 122us) server_negotiation.cc:407] Connection header received
0620 09:26:31.044871 (+ 991us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:31.044872 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:31.044886 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:31.044920 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:31.045781 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:31.046509 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:31.053072 (+ 6563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:31.053427 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:31.053474 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:31.056085 (+ 2611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:31.056087 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:31.056087 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:26:31.056095 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:31.058401 (+ 2306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:31.058837 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:31.058838 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:31.058838 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:31.058896 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:31.059102 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:31.059103 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:31.059103 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:31.059218 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:31.059300 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:31.059447 (+ 147us) server_negotiation.cc:299] Negotiation successful
0620 09:26:31.059549 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 09:26:31.059657 23625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:31.043621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:40995 (local address 127.16.43.129:53467)
0620 09:26:31.043820 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:31.043831 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:31.044707 (+ 876us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:31.044991 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:31.044996 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:31.045454 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:31.045654 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:31.045660 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:31.046626 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:31.046627 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:31.052951 (+ 6324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:31.052954 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:31.053837 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:31.053838 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:31.053912 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:31.054316 (+ 404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:26:31.054321 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:26:31.055964 (+ 1643us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:26:31.058593 (+ 2629us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:31.058594 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:31.058594 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:31.058753 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:31.058973 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:31.058974 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:31.058974 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:31.059029 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:31.059292 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:26:31.059293 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:26:31.059334 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 09:26:31.059422 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 09:26:31.059526 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"mutex_wait_us":9,"thread_start_us":87,"threads_started":1}
I20260620 09:26:31.059937 23620 heartbeater.cc:347] Connected to a master server at 127.16.43.190:40995
I20260620 09:26:31.059983 23620 heartbeater.cc:464] Registering TS with master...
I20260620 09:26:31.060041 23620 heartbeater.cc:511] Master 127.16.43.190:40995 requested a full tablet report, sending...
I20260620 09:26:31.060287 22657 ts_manager.cc:194] Re-registered known tserver with Master: 625889083e23420fb414432cf88b0b51 (127.16.43.129:37091)
I20260620 09:26:31.060693 22657 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.43.129@KRBTEST.COM'} at 127.16.43.129:53467
W20260620 09:26:31.152643 23627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:26:31.152894 23627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:26:31.152949 23627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:26:31.154347 23627 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:26:31.154425 23627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:26:31.154470 23627 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.43.130
I20260620 09:26:31.156215 23627 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:34583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/krb5kdc/oryx_127.16.43.130.keytab
--principal=oryx/127.16.43.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.43.130:38265
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.43.130
--webserver_port=33639
--tserver_master_addrs=127.16.43.190:40995,127.16.43.189:33891,127.16.43.188:46141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23627
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.43.130
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:26:31.156509 23627 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:26:31.156858 23627 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:26:31.162232 23632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:26:31.162276 23633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:26:31.162262 23635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:26:31.162420 23627 server_base.cc:1034] running on GCE node
I20260620 09:26:31.162621 23627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:26:31.162873 23627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:26:31.164024 23627 hybrid_clock.cc:648] HybridClock initialized: now 1781947591164011 us; error 29 us; skew 500 ppm
Jun 20 09:26:31 dist-test-slave-706q krb5kdc[22612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947591, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.43.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:26:31.166706 23627 init.cc:377] Logged in from keytab as oryx/127.16.43.130@KRBTEST.COM (short username oryx)
I20260620 09:26:31.167098 23627 webserver.cc:460] Webserver started at http://127.16.43.130:33639/ using document root <none> and password file <none>
I20260620 09:26:31.167352 23627 fs_manager.cc:359] Metadata directory not provided
I20260620 09:26:31.167435 23627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:26:31.168957 23627 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:26:31.169633 23642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:31.169879 23627 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:26:31.169970 23627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/ts-1/wal
uuid: "318c1dc31882444aa5becb0236f255ea"
format_stamp: "Formatted at 2026-06-20 09:26:30 on dist-test-slave-706q"
server_key: "5a51092e1d76c722d7f6ebe5a5aa544d"
server_key_iv: "69e58789845253ae5677de96d2b3b663"
server_key_version: "encryptionkey@0"
I20260620 09:26:31.170357 23627 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:26:31.184024 23627 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:26:31.186664 23627 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:26:31.186841 23627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:26:31.187088 23627 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:26:31.187398 23627 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:26:31.187453 23627 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:31.187497 23627 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:26:31.187606 23627 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:31.193480 23627 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.130:38265
I20260620 09:26:31.193559 23755 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.130:38265 every 8 connection(s)
I20260620 09:26:31.193914 23627 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/ts-1/data/info.pb
I20260620 09:26:31.200453 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 23627
I20260620 09:26:31.200626 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 22987
Jun 20 09:26:31 dist-test-slave-706q krb5kdc[22612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947591, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.43.130@KRBTEST.COM for oryx/127.16.43.189@KRBTEST.COM
Jun 20 09:26:31 dist-test-slave-706q krb5kdc[22612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947591, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.43.130@KRBTEST.COM for oryx/127.16.43.190@KRBTEST.COM
Jun 20 09:26:31 dist-test-slave-706q krb5kdc[22612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947591, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.43.130@KRBTEST.COM for oryx/127.16.43.188@KRBTEST.COM
I20260620 09:26:31.206485 23305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:31.195333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.130:43383 (local address 127.16.43.189:33891)
0620 09:26:31.195381 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:31.195382 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:31.195391 (+ 9us) server_negotiation.cc:407] Connection header received
0620 09:26:31.195839 (+ 448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:31.195840 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:31.195853 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:31.195887 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:31.196631 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:31.197375 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:31.198613 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:31.198921 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:31.198994 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:31.202934 (+ 3940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:31.202936 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:31.202936 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:26:31.202944 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:31.204860 (+ 1916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:31.205388 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:31.205389 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:31.205390 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:31.205453 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:31.205712 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:31.205713 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:31.205713 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:31.205854 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:31.205901 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:31.206295 (+ 394us) server_negotiation.cc:299] Negotiation successful
0620 09:26:31.206382 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 09:26:31.206557 23623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:31.194743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.130:46723 (local address 127.16.43.188:46141)
0620 09:26:31.194821 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:31.194823 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:31.195068 (+ 245us) server_negotiation.cc:407] Connection header received
0620 09:26:31.195838 (+ 770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:31.195840 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:31.195855 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:31.195887 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:31.196609 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:31.197174 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:31.198613 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:31.198921 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:31.198994 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:31.202908 (+ 3914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:31.202911 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:31.202911 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:26:31.202920 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:31.204728 (+ 1808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:31.205351 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:31.205354 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:31.205354 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:31.205416 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:31.205700 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:31.205701 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:31.205701 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:31.205857 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:31.205901 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:31.206307 (+ 406us) server_negotiation.cc:299] Negotiation successful
0620 09:26:31.206408 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 09:26:31.206774 22697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:31.195138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.130:42609 (local address 127.16.43.190:40995)
0620 09:26:31.195177 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:31.195179 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:31.195414 (+ 235us) server_negotiation.cc:407] Connection header received
0620 09:26:31.195813 (+ 399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:31.195814 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:31.195826 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:31.195881 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:31.196601 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:31.197386 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:31.199391 (+ 2005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:31.199723 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:31.199792 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:31.203012 (+ 3220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:31.203014 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:31.203015 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:26:31.203022 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:31.205592 (+ 2570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:31.205965 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:31.205966 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:31.205967 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:31.206022 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:31.206209 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:31.206210 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:31.206210 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:31.206337 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:31.206379 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:31.206573 (+ 194us) server_negotiation.cc:299] Negotiation successful
0620 09:26:31.206666 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 09:26:31.207413 23760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:31.194719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.188:46141 (local address 127.16.43.130:46723)
0620 09:26:31.194908 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:31.194950 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:31.195709 (+ 759us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:31.195959 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:31.195961 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:31.196313 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:31.196493 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:31.196499 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:31.197256 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:31.197257 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:31.198516 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:31.198518 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:31.199380 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:31.199381 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:31.199435 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:31.200023 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:26:31.200028 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:26:31.202792 (+ 2764us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:26:31.204820 (+ 2028us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:31.204828 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:31.204831 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:31.205251 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:31.205488 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:31.205489 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:31.205489 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:31.205632 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:31.205955 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:26:31.205956 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:26:31.206137 (+ 181us) client_negotiation.cc:769] Sending connection context
0620 09:26:31.207044 (+ 907us) client_negotiation.cc:240] Negotiation successful
0620 09:26:31.207224 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"mutex_wait_us":61,"thread_start_us":110,"threads_started":1}
I20260620 09:26:31.207408 23761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:31.195206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.189:33891 (local address 127.16.43.130:43383)
0620 09:26:31.195338 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:31.195348 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:31.195710 (+ 362us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:31.195959 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:31.195961 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:31.196421 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:31.196493 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:31.196499 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:31.197477 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:31.197478 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:31.198516 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:31.198518 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:31.199012 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:31.199013 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:31.199153 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:31.199751 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:26:31.199779 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:26:31.202814 (+ 3035us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:26:31.204958 (+ 2144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:31.204959 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:31.204959 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:31.205274 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:31.205530 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:31.205531 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:31.205531 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:31.205634 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:31.205949 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:26:31.205955 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:26:31.206137 (+ 182us) client_negotiation.cc:769] Sending connection context
0620 09:26:31.206759 (+ 622us) client_negotiation.cc:240] Negotiation successful
0620 09:26:31.207080 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"mutex_wait_us":13,"thread_start_us":60,"threads_started":1}
I20260620 09:26:31.207408 23762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:31.195043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:40995 (local address 127.16.43.130:42609)
0620 09:26:31.195331 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:31.195342 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:31.195721 (+ 379us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:31.195957 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:31.195961 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:31.196296 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:31.196492 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:31.196496 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:31.198634 (+ 2138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:31.198635 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:31.199282 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:31.199284 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:31.199799 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:31.199800 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:31.199867 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:31.200176 (+ 309us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:26:31.200180 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:26:31.202911 (+ 2731us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:26:31.205673 (+ 2762us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:31.205674 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:31.205674 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:31.205881 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:31.206088 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:31.206088 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:31.206089 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:31.206138 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:31.206423 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:26:31.206424 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:26:31.206460 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 09:26:31.207020 (+ 560us) client_negotiation.cc:240] Negotiation successful
0620 09:26:31.207138 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"mutex_wait_us":23,"thread_start_us":63,"threads_started":1}
I20260620 09:26:31.208066 23756 heartbeater.cc:347] Connected to a master server at 127.16.43.188:46141
I20260620 09:26:31.208169 23756 heartbeater.cc:464] Registering TS with master...
I20260620 09:26:31.208362 23756 heartbeater.cc:511] Master 127.16.43.188:46141 requested a full tablet report, sending...
I20260620 09:26:31.208639 23757 heartbeater.cc:347] Connected to a master server at 127.16.43.190:40995
I20260620 09:26:31.208698 23757 heartbeater.cc:464] Registering TS with master...
I20260620 09:26:31.208772 23757 heartbeater.cc:511] Master 127.16.43.190:40995 requested a full tablet report, sending...
I20260620 09:26:31.208848 23440 ts_manager.cc:194] Registered new tserver with Master: 318c1dc31882444aa5becb0236f255ea (127.16.43.130:38265)
I20260620 09:26:31.208899 23758 heartbeater.cc:347] Connected to a master server at 127.16.43.189:33891
I20260620 09:26:31.208931 23758 heartbeater.cc:464] Registering TS with master...
I20260620 09:26:31.208987 23758 heartbeater.cc:511] Master 127.16.43.189:33891 requested a full tablet report, sending...
I20260620 09:26:31.208997 22657 ts_manager.cc:194] Re-registered known tserver with Master: 318c1dc31882444aa5becb0236f255ea (127.16.43.130:38265)
I20260620 09:26:31.209262 23254 ts_manager.cc:194] Registered new tserver with Master: 318c1dc31882444aa5becb0236f255ea (127.16.43.130:38265)
I20260620 09:26:31.209396 22657 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.43.130@KRBTEST.COM'} at 127.16.43.130:42609
I20260620 09:26:31.209923 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.43.131:39089
--local_ip_for_outbound_sockets=127.16.43.131
--tserver_master_addrs=127.16.43.190:40995
--webserver_port=38635
--webserver_interface=127.16.43.131
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/krb5kdc/oryx_127.16.43.131.keytab
--principal=oryx/127.16.43.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.43.148:34583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.16.43.190:40995,127.16.43.189:33891
--tserver_master_addrs=127.16.43.190:40995,127.16.43.189:33891,127.16.43.188:46141 with env {KRB5CCNAME=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:26:31.294337 23763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:26:31.294548 23763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:26:31.294574 23763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:26:31.295761 23763 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:26:31.295800 23763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:26:31.295826 23763 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.43.131
I20260620 09:26:31.297242 23763 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:34583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/krb5kdc/oryx_127.16.43.131.keytab
--principal=oryx/127.16.43.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.43.131:39089
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.43.131
--webserver_port=38635
--tserver_master_addrs=127.16.43.190:40995,127.16.43.189:33891,127.16.43.188:46141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23763
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.43.131
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:26:31.297430 23763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:26:31.297612 23763 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:26:31.302958 23771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:26:31.303009 23769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:26:31.302964 23768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:26:31.303200 23763 server_base.cc:1034] running on GCE node
I20260620 09:26:31.303484 23763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:26:31.303802 23763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:26:31.304968 23763 hybrid_clock.cc:648] HybridClock initialized: now 1781947591304955 us; error 30 us; skew 500 ppm
Jun 20 09:26:31 dist-test-slave-706q krb5kdc[22612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947591, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.43.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:26:31.307751 23763 init.cc:377] Logged in from keytab as oryx/127.16.43.131@KRBTEST.COM (short username oryx)
I20260620 09:26:31.308214 23763 webserver.cc:460] Webserver started at http://127.16.43.131:38635/ using document root <none> and password file <none>
I20260620 09:26:31.308455 23763 fs_manager.cc:359] Metadata directory not provided
I20260620 09:26:31.308534 23763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:26:31.310140 23763 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:26:31.310823 23778 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:31.311067 23763 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:26:31.311163 23763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/ts-2/wal
uuid: "3a91f37200fc4719b1d4853f3fbaefe3"
format_stamp: "Formatted at 2026-06-20 09:26:30 on dist-test-slave-706q"
server_key: "8a32481ced996d71f396f6feda360e65"
server_key_iv: "61a1d007146cbede6f3521b91b4bb174"
server_key_version: "encryptionkey@0"
I20260620 09:26:31.311616 23763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:26:31.315603 23623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:31.308980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36184 (local address 127.16.43.188:46141)
0620 09:26:31.309043 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:31.309046 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:31.309057 (+ 11us) server_negotiation.cc:407] Connection header received
0620 09:26:31.309137 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:31.309138 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:31.309149 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:31.309200 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:31.309527 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:31.310345 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:31.311274 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:31.311461 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:31.311517 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:31.312741 (+ 1224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:31.312743 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:31.312744 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:26:31.312750 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:31.314352 (+ 1602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:31.314730 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:31.314731 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:31.314732 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:31.314781 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:31.315022 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:31.315024 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:31.315024 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:31.315166 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:31.315237 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:31.315404 (+ 167us) server_negotiation.cc:299] Negotiation successful
0620 09:26:31.315470 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 09:26:31.315632 23214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:31.308909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.188:46141 (local address 127.0.0.1:36184)
0620 09:26:31.308973 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:31.308983 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:31.309055 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:31.309230 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:31.309231 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:31.309394 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:31.309445 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:31.309448 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:31.310440 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:31.310442 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:31.311154 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:31.311159 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:31.311563 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:31.311564 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:31.311626 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:31.312234 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:26:31.312240 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:26:31.312651 (+ 411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:26:31.314435 (+ 1784us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:31.314436 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:31.314436 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:31.314652 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:31.314851 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:31.314851 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:31.314852 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:31.314922 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:31.315251 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:26:31.315252 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:26:31.315304 (+ 52us) client_negotiation.cc:769] Sending connection context
0620 09:26:31.315383 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 09:26:31.315489 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260620 09:26:31.315944 23450 raft_consensus.cc:1203] T 00000000000000000000000000000000 P fc318f0d33f1422abfc6e301edf09951 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260620 09:26:31.316149 23479 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc318f0d33f1422abfc6e301edf09951 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d2095405b6a4278a42cc71c3c787e3e. Latest consensus state: current_term: 1 leader_uuid: "1d2095405b6a4278a42cc71c3c787e3e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1d2095405b6a4278a42cc71c3c787e3e" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 40995 } } peers { permanent_uuid: "13835b763e164a3f9232bac524dd0bac" member_type: VOTER last_known_addr { host: "127.16.43.189" port: 33891 } attrs { promote: false } } peers { permanent_uuid: "fc318f0d33f1422abfc6e301edf09951" member_type: NON_VOTER last_known_addr { host: "127.16.43.188" port: 46141 } attrs { promote: true } } }
I20260620 09:26:31.316244 23479 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc318f0d33f1422abfc6e301edf09951 [sys.catalog]: This master's current role is: LEARNER
I20260620 09:26:31.336545 23763 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:26:31.338873 23763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:26:31.339036 23763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:26:31.339313 23763 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:26:31.339705 23763 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:26:31.339766 23763 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:31.339811 23763 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:26:31.339838 23763 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:31.345758 23763 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.131:39089
I20260620 09:26:31.345817 23891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.131:39089 every 8 connection(s)
I20260620 09:26:31.346206 23763 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/ts-2/data/info.pb
Jun 20 09:26:31 dist-test-slave-706q krb5kdc[22612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947591, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.43.131@KRBTEST.COM for oryx/127.16.43.188@KRBTEST.COM
Jun 20 09:26:31 dist-test-slave-706q krb5kdc[22612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947591, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.43.131@KRBTEST.COM for oryx/127.16.43.190@KRBTEST.COM
Jun 20 09:26:31 dist-test-slave-706q krb5kdc[22612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947591, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.43.131@KRBTEST.COM for oryx/127.16.43.189@KRBTEST.COM
I20260620 09:26:31.355249 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 23763
I20260620 09:26:31.357730 23623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:31.347001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.131:49533 (local address 127.16.43.188:46141)
0620 09:26:31.347049 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:31.347051 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:31.347229 (+ 178us) server_negotiation.cc:407] Connection header received
0620 09:26:31.347986 (+ 757us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:31.347987 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:31.348001 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:31.348044 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:31.348813 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:31.349576 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:31.351491 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:31.351786 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:31.351857 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:31.354157 (+ 2300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:31.354159 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:31.354159 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:26:31.354168 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:31.355966 (+ 1798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:31.356482 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:31.356483 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:31.356484 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:31.356540 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:31.356837 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:31.356837 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:31.356838 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:31.356984 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:31.357025 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:31.357531 (+ 506us) server_negotiation.cc:299] Negotiation successful
0620 09:26:31.357615 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 09:26:31.357878 23896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:31.346946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.188:46141 (local address 127.16.43.131:49533)
0620 09:26:31.347115 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:31.347159 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:31.347870 (+ 711us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:31.348114 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:31.348125 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:31.348486 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:31.348711 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:31.348715 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:31.350801 (+ 2086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:31.350801 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:31.351405 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:31.351407 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:31.351864 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:31.351865 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:31.351905 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:31.352273 (+ 368us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:26:31.352279 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:26:31.354044 (+ 1765us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:26:31.356057 (+ 2013us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:31.356066 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:31.356068 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:31.356390 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:31.356617 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:31.356618 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:31.356618 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:31.356756 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:31.357076 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:26:31.357082 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:26:31.357271 (+ 189us) client_negotiation.cc:769] Sending connection context
0620 09:26:31.357392 (+ 121us) client_negotiation.cc:240] Negotiation successful
0620 09:26:31.357640 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"mutex_wait_us":1,"thread_start_us":98,"threads_started":1}
I20260620 09:26:31.358331 23892 heartbeater.cc:347] Connected to a master server at 127.16.43.188:46141
I20260620 09:26:31.358436 23892 heartbeater.cc:464] Registering TS with master...
I20260620 09:26:31.358500 23305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:31.347483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.131:36311 (local address 127.16.43.189:33891)
0620 09:26:31.347552 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:31.347554 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:31.347566 (+ 12us) server_negotiation.cc:407] Connection header received
0620 09:26:31.347987 (+ 421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:31.347988 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:31.348001 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:31.348034 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:31.348873 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:31.349577 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:31.350802 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:31.351105 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:31.351182 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:31.355129 (+ 3947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:31.355131 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:31.355131 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:26:31.355139 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:31.356838 (+ 1699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:31.357320 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:31.357322 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:31.357322 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:31.357380 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:31.357727 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:31.357728 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:31.357728 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:31.357851 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:31.357892 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:31.358335 (+ 443us) server_negotiation.cc:299] Negotiation successful
0620 09:26:31.358413 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 09:26:31.358626 23892 heartbeater.cc:511] Master 127.16.43.188:46141 requested a full tablet report, sending...
I20260620 09:26:31.359009 23898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:31.347410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.189:33891 (local address 127.16.43.131:36311)
0620 09:26:31.347505 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:31.347515 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:31.347888 (+ 373us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:31.348511 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:31.348512 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:31.348665 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:31.348806 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:31.348808 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:31.349661 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:31.349662 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:31.350698 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:31.350701 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:31.351578 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:31.351579 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:31.351638 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:31.352248 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:26:31.352252 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:26:31.355060 (+ 2808us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:26:31.356919 (+ 1859us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:31.356920 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:31.356920 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:31.357235 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:31.357498 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:31.357499 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:31.357499 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:31.357632 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:31.358170 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:26:31.358171 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:26:31.358216 (+ 45us) client_negotiation.cc:769] Sending connection context
0620 09:26:31.358813 (+ 597us) client_negotiation.cc:240] Negotiation successful
0620 09:26:31.358922 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80,"mutex_wait_us":10,"thread_start_us":55,"threads_started":1}
I20260620 09:26:31.359068 23440 ts_manager.cc:194] Registered new tserver with Master: 3a91f37200fc4719b1d4853f3fbaefe3 (127.16.43.131:39089)
I20260620 09:26:31.359413 23894 heartbeater.cc:347] Connected to a master server at 127.16.43.189:33891
I20260620 09:26:31.359468 23894 heartbeater.cc:464] Registering TS with master...
I20260620 09:26:31.359551 23894 heartbeater.cc:511] Master 127.16.43.189:33891 requested a full tablet report, sending...
I20260620 09:26:31.359823 23254 ts_manager.cc:194] Registered new tserver with Master: 3a91f37200fc4719b1d4853f3fbaefe3 (127.16.43.131:39089)
I20260620 09:26:31.359858 22697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:31.347333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.131:58257 (local address 127.16.43.190:40995)
0620 09:26:31.347521 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:31.347546 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:31.347557 (+ 11us) server_negotiation.cc:407] Connection header received
0620 09:26:31.347988 (+ 431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:31.347990 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:31.348001 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:31.348034 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:31.348827 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:31.349351 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:31.350805 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:31.351105 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:31.351186 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:31.354573 (+ 3387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:31.354574 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:31.354575 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:26:31.354583 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:31.357705 (+ 3122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:31.358179 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:31.358180 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:31.358180 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:31.358243 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:31.358786 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:31.358787 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:31.358787 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:31.358932 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:31.358997 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:31.359644 (+ 647us) server_negotiation.cc:299] Negotiation successful
0620 09:26:31.359738 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158}
I20260620 09:26:31.359416 23897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:31.347257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:40995 (local address 127.16.43.131:58257)
0620 09:26:31.347382 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:31.347392 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:31.347888 (+ 496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:31.348121 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:31.348122 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:31.348495 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:31.348713 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:31.348720 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:31.349453 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:31.349453 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:31.350698 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:31.350701 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:31.351196 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:31.351197 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:31.351342 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:31.352233 (+ 891us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:26:31.352243 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:26:31.354454 (+ 2211us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:26:31.357797 (+ 3343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:31.357798 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:31.357799 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:31.358085 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:31.358320 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:31.358321 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:31.358322 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:31.358407 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:31.359027 (+ 620us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:26:31.359028 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:26:31.359079 (+ 51us) client_negotiation.cc:769] Sending connection context
0620 09:26:31.359149 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 09:26:31.359262 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"mutex_wait_us":31,"thread_start_us":67,"threads_started":1}
I20260620 09:26:31.360455 23893 heartbeater.cc:347] Connected to a master server at 127.16.43.190:40995
I20260620 09:26:31.360502 23893 heartbeater.cc:464] Registering TS with master...
I20260620 09:26:31.360556 23893 heartbeater.cc:511] Master 127.16.43.190:40995 requested a full tablet report, sending...
I20260620 09:26:31.360846 22657 ts_manager.cc:194] Re-registered known tserver with Master: 3a91f37200fc4719b1d4853f3fbaefe3 (127.16.43.131:39089)
I20260620 09:26:31.361316 22657 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.43.131@KRBTEST.COM'} at 127.16.43.131:58257
I20260620 09:26:31.366973 23623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:31.357094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36196 (local address 127.16.43.188:46141)
0620 09:26:31.358026 (+ 932us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:31.358028 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:31.359350 (+ 1322us) server_negotiation.cc:407] Connection header received
0620 09:26:31.359443 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:31.359444 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:31.359634 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:31.359669 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:31.360428 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:31.361170 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:31.362192 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:31.362477 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:31.362543 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:31.363967 (+ 1424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:31.363968 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:31.363969 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:26:31.363976 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:31.365693 (+ 1717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:31.366109 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:31.366110 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:31.366111 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:31.366164 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:31.366396 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:31.366397 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:31.366398 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:31.366531 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:31.366589 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:31.366777 (+ 188us) server_negotiation.cc:299] Negotiation successful
0620 09:26:31.366853 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":817,"thread_start_us":52,"threads_started":1}
I20260620 09:26:31.367090 23305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:31.357490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42460 (local address 127.16.43.189:33891)
0620 09:26:31.358568 (+ 1078us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:31.358570 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:31.358577 (+ 7us) server_negotiation.cc:407] Connection header received
0620 09:26:31.358596 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:31.358596 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:31.358606 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:31.358661 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:31.360263 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:31.360994 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:31.361909 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:31.362171 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:31.362242 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:31.363926 (+ 1684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:31.363928 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:31.363928 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:26:31.363936 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:31.365669 (+ 1733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:31.366191 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:31.366192 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:31.366193 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:31.366251 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:31.366472 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:31.366473 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:31.366473 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:31.366586 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:31.366692 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:31.366902 (+ 210us) server_negotiation.cc:299] Negotiation successful
0620 09:26:31.366983 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1061,"thread_start_us":103,"threads_started":1}
I20260620 09:26:31.368739 23903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:31.356346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40588 (local address 127.16.43.190:40995)
0620 09:26:31.358238 (+ 1892us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:31.358240 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:31.358353 (+ 113us) server_negotiation.cc:407] Connection header received
0620 09:26:31.358639 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:31.358640 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:31.358654 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:31.358704 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:31.361951 (+ 3247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:31.362670 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:31.363499 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:31.363763 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:31.363847 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:31.365288 (+ 1441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:31.365289 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:31.365289 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:26:31.365296 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:31.367594 (+ 2298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:31.367954 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:31.367956 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:31.367956 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:31.368013 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:31.368221 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:31.368222 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:31.368222 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:31.368337 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:31.368443 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:31.368542 (+ 99us) server_negotiation.cc:299] Negotiation successful
0620 09:26:31.368624 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1855,"thread_start_us":87,"threads_started":1}
I20260620 09:26:31.370373 22657 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40588:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 09:26:31.371999 22657 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 09:26:31.372711 23215 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d2095405b6a4278a42cc71c3c787e3e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1d2095405b6a4278a42cc71c3c787e3e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1d2095405b6a4278a42cc71c3c787e3e" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 40995 } } peers { permanent_uuid: "13835b763e164a3f9232bac524dd0bac" member_type: VOTER last_known_addr { host: "127.16.43.189" port: 33891 } attrs { promote: false } } peers { permanent_uuid: "fc318f0d33f1422abfc6e301edf09951" member_type: NON_VOTER last_known_addr { host: "127.16.43.188" port: 46141 } attrs { promote: true } } }
I20260620 09:26:31.372830 23215 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d2095405b6a4278a42cc71c3c787e3e [sys.catalog]: This master's current role is: LEADER
I20260620 09:26:31.374145 23215 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 1d2095405b6a4278a42cc71c3c787e3e: attempting to promote NON_VOTER fc318f0d33f1422abfc6e301edf09951 to VOTER
I20260620 09:26:31.374287 23215 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1d2095405b6a4278a42cc71c3c787e3e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1d2095405b6a4278a42cc71c3c787e3e" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 40995 } } peers { permanent_uuid: "13835b763e164a3f9232bac524dd0bac" member_type: VOTER last_known_addr { host: "127.16.43.189" port: 33891 } attrs { promote: false } } peers { permanent_uuid: "fc318f0d33f1422abfc6e301edf09951" member_type: VOTER last_known_addr { host: "127.16.43.188" port: 46141 } attrs { promote: false } }
I20260620 09:26:31.374778 23264 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 13835b763e164a3f9232bac524dd0bac [term 1 FOLLOWER]: Refusing update from remote peer 1d2095405b6a4278a42cc71c3c787e3e: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260620 09:26:31.374941 23402 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1d2095405b6a4278a42cc71c3c787e3e [LEADER]: Connected to new peer: Peer: permanent_uuid: "13835b763e164a3f9232bac524dd0bac" member_type: VOTER last_known_addr { host: "127.16.43.189" port: 33891 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260620 09:26:31.375095 23450 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fc318f0d33f1422abfc6e301edf09951 [term 1 LEARNER]: Refusing update from remote peer 1d2095405b6a4278a42cc71c3c787e3e: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260620 09:26:31.375370 23402 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1d2095405b6a4278a42cc71c3c787e3e [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc318f0d33f1422abfc6e301edf09951" member_type: VOTER last_known_addr { host: "127.16.43.188" port: 46141 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260620 09:26:31.375487 23216 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1d2095405b6a4278a42cc71c3c787e3e [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, fc318f0d33f1422abfc6e301edf09951 (127.16.43.188) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1d2095405b6a4278a42cc71c3c787e3e" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 40995 } } peers { permanent_uuid: "13835b763e164a3f9232bac524dd0bac" member_type: VOTER last_known_addr { host: "127.16.43.189" port: 33891 } attrs { promote: false } } peers { permanent_uuid: "fc318f0d33f1422abfc6e301edf09951" member_type: VOTER last_known_addr { host: "127.16.43.188" port: 46141 } attrs { promote: false } } }
I20260620 09:26:31.375680 23264 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 13835b763e164a3f9232bac524dd0bac [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, fc318f0d33f1422abfc6e301edf09951 (127.16.43.188) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1d2095405b6a4278a42cc71c3c787e3e" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 40995 } } peers { permanent_uuid: "13835b763e164a3f9232bac524dd0bac" member_type: VOTER last_known_addr { host: "127.16.43.189" port: 33891 } attrs { promote: false } } peers { permanent_uuid: "fc318f0d33f1422abfc6e301edf09951" member_type: VOTER last_known_addr { host: "127.16.43.188" port: 46141 } attrs { promote: false } } }
I20260620 09:26:31.376237 23450 raft_consensus.cc:2932] T 00000000000000000000000000000000 P fc318f0d33f1422abfc6e301edf09951 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, fc318f0d33f1422abfc6e301edf09951 (127.16.43.188) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1d2095405b6a4278a42cc71c3c787e3e" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 40995 } } peers { permanent_uuid: "13835b763e164a3f9232bac524dd0bac" member_type: VOTER last_known_addr { host: "127.16.43.189" port: 33891 } attrs { promote: false } } peers { permanent_uuid: "fc318f0d33f1422abfc6e301edf09951" member_type: VOTER last_known_addr { host: "127.16.43.188" port: 46141 } attrs { promote: false } } }
I20260620 09:26:31.376518 23216 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d2095405b6a4278a42cc71c3c787e3e [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "1d2095405b6a4278a42cc71c3c787e3e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1d2095405b6a4278a42cc71c3c787e3e" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 40995 } } peers { permanent_uuid: "13835b763e164a3f9232bac524dd0bac" member_type: VOTER last_known_addr { host: "127.16.43.189" port: 33891 } attrs { promote: false } } peers { permanent_uuid: "fc318f0d33f1422abfc6e301edf09951" member_type: VOTER last_known_addr { host: "127.16.43.188" port: 46141 } attrs { promote: false } } }
I20260620 09:26:31.376591 23216 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d2095405b6a4278a42cc71c3c787e3e [sys.catalog]: This master's current role is: LEADER
I20260620 09:26:31.376657 23216 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d2095405b6a4278a42cc71c3c787e3e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1d2095405b6a4278a42cc71c3c787e3e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1d2095405b6a4278a42cc71c3c787e3e" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 40995 } } peers { permanent_uuid: "13835b763e164a3f9232bac524dd0bac" member_type: VOTER last_known_addr { host: "127.16.43.189" port: 33891 } attrs { promote: false } } peers { permanent_uuid: "fc318f0d33f1422abfc6e301edf09951" member_type: VOTER last_known_addr { host: "127.16.43.188" port: 46141 } attrs { promote: false } } }
I20260620 09:26:31.376725 23216 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d2095405b6a4278a42cc71c3c787e3e [sys.catalog]: This master's current role is: LEADER
I20260620 09:26:31.376679 23293 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13835b763e164a3f9232bac524dd0bac [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1d2095405b6a4278a42cc71c3c787e3e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1d2095405b6a4278a42cc71c3c787e3e" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 40995 } } peers { permanent_uuid: "13835b763e164a3f9232bac524dd0bac" member_type: VOTER last_known_addr { host: "127.16.43.189" port: 33891 } attrs { promote: false } } peers { permanent_uuid: "fc318f0d33f1422abfc6e301edf09951" member_type: VOTER last_known_addr { host: "127.16.43.188" port: 46141 } attrs { promote: false } } }
I20260620 09:26:31.376756 23293 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13835b763e164a3f9232bac524dd0bac [sys.catalog]: This master's current role is: FOLLOWER
I20260620 09:26:31.376784 23216 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d2095405b6a4278a42cc71c3c787e3e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1d2095405b6a4278a42cc71c3c787e3e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1d2095405b6a4278a42cc71c3c787e3e" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 40995 } } peers { permanent_uuid: "13835b763e164a3f9232bac524dd0bac" member_type: VOTER last_known_addr { host: "127.16.43.189" port: 33891 } attrs { promote: false } } peers { permanent_uuid: "fc318f0d33f1422abfc6e301edf09951" member_type: VOTER last_known_addr { host: "127.16.43.188" port: 46141 } attrs { promote: false } } }
I20260620 09:26:31.376821 23216 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d2095405b6a4278a42cc71c3c787e3e [sys.catalog]: This master's current role is: LEADER
I20260620 09:26:31.377342 23479 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc318f0d33f1422abfc6e301edf09951 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1d2095405b6a4278a42cc71c3c787e3e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1d2095405b6a4278a42cc71c3c787e3e" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 40995 } } peers { permanent_uuid: "13835b763e164a3f9232bac524dd0bac" member_type: VOTER last_known_addr { host: "127.16.43.189" port: 33891 } attrs { promote: false } } peers { permanent_uuid: "fc318f0d33f1422abfc6e301edf09951" member_type: VOTER last_known_addr { host: "127.16.43.188" port: 46141 } attrs { promote: false } } }
I20260620 09:26:31.377437 23479 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc318f0d33f1422abfc6e301edf09951 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 09:26:31.381474 23920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:31.377690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33106 (local address 127.16.43.130:38265)
0620 09:26:31.377854 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:31.377859 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:31.378086 (+ 227us) server_negotiation.cc:407] Connection header received
0620 09:26:31.378169 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:31.378171 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:31.378188 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:31.378264 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:26:31.378446 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:31.379478 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:31.380392 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:31.381100 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:31.381193 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:31.381279 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:31.381359 (+ 80us) server_negotiation.cc:299] Negotiation successful
0620 09:26:31.381394 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":92,"threads_started":1}
I20260620 09:26:31.381471 23917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:31.377309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.130:38265 (local address 127.0.0.1:33106)
0620 09:26:31.378027 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:31.378037 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:31.378090 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:31.378337 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:31.378338 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:31.378340 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:26:31.378373 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:31.378375 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:31.379570 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:31.379571 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:31.380298 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:31.380300 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:31.381203 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:31.381205 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:31.381280 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:31.381286 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 09:26:31.381344 (+ 58us) client_negotiation.cc:240] Negotiation successful
0620 09:26:31.381387 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":699,"thread_start_us":82,"threads_started":1}
I20260620 09:26:31.381798 23214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:31.377259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.131:39089 (local address 127.0.0.1:59888)
0620 09:26:31.377381 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:31.377393 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:31.377435 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:31.378040 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:31.378041 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:31.378043 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:26:31.378095 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:31.378097 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:31.379471 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:31.379472 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:31.380656 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:31.380659 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:31.381595 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:31.381596 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:31.381657 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:31.381661 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 09:26:31.381693 (+ 32us) client_negotiation.cc:240] Negotiation successful
0620 09:26:31.381731 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"mutex_wait_us":37}
I20260620 09:26:31.382323 23918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:31.377522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59888 (local address 127.16.43.131:39089)
0620 09:26:31.377836 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:31.377841 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:31.377855 (+ 14us) server_negotiation.cc:407] Connection header received
0620 09:26:31.377897 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:31.377899 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:31.377916 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:31.378174 (+ 258us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:26:31.378252 (+ 78us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:31.379403 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:31.380762 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:31.381492 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:31.381925 (+ 433us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:31.382011 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:31.382037 (+ 26us) server_negotiation.cc:299] Negotiation successful
0620 09:26:31.382080 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":97,"threads_started":1}
I20260620 09:26:31.382495 23690 tablet_service.cc:1467] Processing CreateTablet for tablet f9641b40654a4c1199149f3c742b8595 (DEFAULT_TABLE table=test-table [id=84918ab846f0469d9b189b557ecfba46]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:26:31.382900 23690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9641b40654a4c1199149f3c742b8595. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:26:31.382995 23826 tablet_service.cc:1467] Processing CreateTablet for tablet f9641b40654a4c1199149f3c742b8595 (DEFAULT_TABLE table=test-table [id=84918ab846f0469d9b189b557ecfba46]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:26:31.383325 23826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9641b40654a4c1199149f3c742b8595. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:26:31.386457 23921 tablet_bootstrap.cc:492] T f9641b40654a4c1199149f3c742b8595 P 318c1dc31882444aa5becb0236f255ea: Bootstrap starting.
I20260620 09:26:31.386595 23916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:31.377494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.129:37091 (local address 127.0.0.1:34000)
0620 09:26:31.378166 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:31.378172 (+ 6us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:31.378232 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:31.378480 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:31.378481 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:31.378483 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:26:31.378528 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:31.378530 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:31.379390 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:31.379391 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:31.380670 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:31.380672 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:31.381954 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:31.381955 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:31.386357 (+ 4402us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:31.386365 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 09:26:31.386426 (+ 61us) client_negotiation.cc:240] Negotiation successful
0620 09:26:31.386487 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":662,"mutex_wait_us":35,"thread_start_us":66,"threads_started":1}
I20260620 09:26:31.386830 23922 tablet_bootstrap.cc:492] T f9641b40654a4c1199149f3c742b8595 P 3a91f37200fc4719b1d4853f3fbaefe3: Bootstrap starting.
I20260620 09:26:31.387189 23919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:31.377610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34000 (local address 127.16.43.129:37091)
0620 09:26:31.377806 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:31.377812 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:31.378239 (+ 427us) server_negotiation.cc:407] Connection header received
0620 09:26:31.378300 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:31.378302 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:31.378338 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:31.378410 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:26:31.378604 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:31.379290 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:31.380764 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:31.381493 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:31.381566 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:31.386984 (+ 5418us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:31.387025 (+ 41us) server_negotiation.cc:299] Negotiation successful
0620 09:26:31.387087 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":89,"threads_started":1}
I20260620 09:26:31.387362 23921 tablet_bootstrap.cc:654] T f9641b40654a4c1199149f3c742b8595 P 318c1dc31882444aa5becb0236f255ea: Neither blocks nor log segments found. Creating new log.
I20260620 09:26:31.387413 23922 tablet_bootstrap.cc:654] T f9641b40654a4c1199149f3c742b8595 P 3a91f37200fc4719b1d4853f3fbaefe3: Neither blocks nor log segments found. Creating new log.
I20260620 09:26:31.387758 23922 log.cc:824] T f9641b40654a4c1199149f3c742b8595 P 3a91f37200fc4719b1d4853f3fbaefe3: Log is configured to *not* fsync() on all Append() calls
I20260620 09:26:31.387764 23921 log.cc:824] T f9641b40654a4c1199149f3c742b8595 P 318c1dc31882444aa5becb0236f255ea: Log is configured to *not* fsync() on all Append() calls
I20260620 09:26:31.387862 23553 tablet_service.cc:1467] Processing CreateTablet for tablet f9641b40654a4c1199149f3c742b8595 (DEFAULT_TABLE table=test-table [id=84918ab846f0469d9b189b557ecfba46]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:26:31.388250 23553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9641b40654a4c1199149f3c742b8595. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:26:31.388634 23922 tablet_bootstrap.cc:492] T f9641b40654a4c1199149f3c742b8595 P 3a91f37200fc4719b1d4853f3fbaefe3: No bootstrap required, opened a new log
I20260620 09:26:31.388789 23921 tablet_bootstrap.cc:492] T f9641b40654a4c1199149f3c742b8595 P 318c1dc31882444aa5becb0236f255ea: No bootstrap required, opened a new log
I20260620 09:26:31.388708 23922 ts_tablet_manager.cc:1397] T f9641b40654a4c1199149f3c742b8595 P 3a91f37200fc4719b1d4853f3fbaefe3: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 09:26:31.388856 23921 ts_tablet_manager.cc:1397] T f9641b40654a4c1199149f3c742b8595 P 318c1dc31882444aa5becb0236f255ea: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260620 09:26:31.390944 23925 tablet_bootstrap.cc:492] T f9641b40654a4c1199149f3c742b8595 P 625889083e23420fb414432cf88b0b51: Bootstrap starting.
I20260620 09:26:31.390873 23922 raft_consensus.cc:348] T f9641b40654a4c1199149f3c742b8595 P 3a91f37200fc4719b1d4853f3fbaefe3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a91f37200fc4719b1d4853f3fbaefe3" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 39089 } } peers { permanent_uuid: "318c1dc31882444aa5becb0236f255ea" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 38265 } } peers { permanent_uuid: "625889083e23420fb414432cf88b0b51" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 37091 } }
I20260620 09:26:31.391165 23922 raft_consensus.cc:374] T f9641b40654a4c1199149f3c742b8595 P 3a91f37200fc4719b1d4853f3fbaefe3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:26:31.391223 23922 raft_consensus.cc:729] T f9641b40654a4c1199149f3c742b8595 P 3a91f37200fc4719b1d4853f3fbaefe3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a91f37200fc4719b1d4853f3fbaefe3, State: Initialized, Role: FOLLOWER
I20260620 09:26:31.391351 23922 consensus_queue.cc:260] T f9641b40654a4c1199149f3c742b8595 P 3a91f37200fc4719b1d4853f3fbaefe3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a91f37200fc4719b1d4853f3fbaefe3" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 39089 } } peers { permanent_uuid: "318c1dc31882444aa5becb0236f255ea" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 38265 } } peers { permanent_uuid: "625889083e23420fb414432cf88b0b51" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 37091 } }
I20260620 09:26:31.391666 23922 ts_tablet_manager.cc:1428] T f9641b40654a4c1199149f3c742b8595 P 3a91f37200fc4719b1d4853f3fbaefe3: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260620 09:26:31.391885 23921 raft_consensus.cc:348] T f9641b40654a4c1199149f3c742b8595 P 318c1dc31882444aa5becb0236f255ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a91f37200fc4719b1d4853f3fbaefe3" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 39089 } } peers { permanent_uuid: "318c1dc31882444aa5becb0236f255ea" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 38265 } } peers { permanent_uuid: "625889083e23420fb414432cf88b0b51" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 37091 } }
I20260620 09:26:31.391991 23893 heartbeater.cc:503] Master 127.16.43.190:40995 was elected leader, sending a full tablet report...
I20260620 09:26:31.392032 23921 raft_consensus.cc:374] T f9641b40654a4c1199149f3c742b8595 P 318c1dc31882444aa5becb0236f255ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:26:31.392076 23921 raft_consensus.cc:729] T f9641b40654a4c1199149f3c742b8595 P 318c1dc31882444aa5becb0236f255ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 318c1dc31882444aa5becb0236f255ea, State: Initialized, Role: FOLLOWER
I20260620 09:26:31.392135 23925 tablet_bootstrap.cc:654] T f9641b40654a4c1199149f3c742b8595 P 625889083e23420fb414432cf88b0b51: Neither blocks nor log segments found. Creating new log.
I20260620 09:26:31.392194 23921 consensus_queue.cc:260] T f9641b40654a4c1199149f3c742b8595 P 318c1dc31882444aa5becb0236f255ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a91f37200fc4719b1d4853f3fbaefe3" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 39089 } } peers { permanent_uuid: "318c1dc31882444aa5becb0236f255ea" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 38265 } } peers { permanent_uuid: "625889083e23420fb414432cf88b0b51" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 37091 } }
I20260620 09:26:31.392442 23925 log.cc:824] T f9641b40654a4c1199149f3c742b8595 P 625889083e23420fb414432cf88b0b51: Log is configured to *not* fsync() on all Append() calls
I20260620 09:26:31.392803 23921 ts_tablet_manager.cc:1428] T f9641b40654a4c1199149f3c742b8595 P 318c1dc31882444aa5becb0236f255ea: Time spent starting tablet: real 0.004s user 0.002s sys 0.002s
I20260620 09:26:31.392938 23757 heartbeater.cc:503] Master 127.16.43.190:40995 was elected leader, sending a full tablet report...
I20260620 09:26:31.393209 23925 tablet_bootstrap.cc:492] T f9641b40654a4c1199149f3c742b8595 P 625889083e23420fb414432cf88b0b51: No bootstrap required, opened a new log
I20260620 09:26:31.393270 23925 ts_tablet_manager.cc:1397] T f9641b40654a4c1199149f3c742b8595 P 625889083e23420fb414432cf88b0b51: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 09:26:31.395071 23925 raft_consensus.cc:348] T f9641b40654a4c1199149f3c742b8595 P 625889083e23420fb414432cf88b0b51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a91f37200fc4719b1d4853f3fbaefe3" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 39089 } } peers { permanent_uuid: "318c1dc31882444aa5becb0236f255ea" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 38265 } } peers { permanent_uuid: "625889083e23420fb414432cf88b0b51" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 37091 } }
I20260620 09:26:31.395229 23925 raft_consensus.cc:374] T f9641b40654a4c1199149f3c742b8595 P 625889083e23420fb414432cf88b0b51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:26:31.395277 23925 raft_consensus.cc:729] T f9641b40654a4c1199149f3c742b8595 P 625889083e23420fb414432cf88b0b51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 625889083e23420fb414432cf88b0b51, State: Initialized, Role: FOLLOWER
I20260620 09:26:31.395385 23925 consensus_queue.cc:260] T f9641b40654a4c1199149f3c742b8595 P 625889083e23420fb414432cf88b0b51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a91f37200fc4719b1d4853f3fbaefe3" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 39089 } } peers { permanent_uuid: "318c1dc31882444aa5becb0236f255ea" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 38265 } } peers { permanent_uuid: "625889083e23420fb414432cf88b0b51" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 37091 } }
I20260620 09:26:31.395653 23925 ts_tablet_manager.cc:1428] T f9641b40654a4c1199149f3c742b8595 P 625889083e23420fb414432cf88b0b51: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20260620 09:26:31.395916 23620 heartbeater.cc:503] Master 127.16.43.190:40995 was elected leader, sending a full tablet report...
I20260620 09:26:31.402568 23928 raft_consensus.cc:482] T f9641b40654a4c1199149f3c742b8595 P 318c1dc31882444aa5becb0236f255ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:26:31.402689 23928 raft_consensus.cc:504] T f9641b40654a4c1199149f3c742b8595 P 318c1dc31882444aa5becb0236f255ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a91f37200fc4719b1d4853f3fbaefe3" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 39089 } } peers { permanent_uuid: "318c1dc31882444aa5becb0236f255ea" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 38265 } } peers { permanent_uuid: "625889083e23420fb414432cf88b0b51" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 37091 } }
I20260620 09:26:31.403105 23928 leader_election.cc:288] T f9641b40654a4c1199149f3c742b8595 P 318c1dc31882444aa5becb0236f255ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3a91f37200fc4719b1d4853f3fbaefe3 (127.16.43.131:39089), 625889083e23420fb414432cf88b0b51 (127.16.43.129:37091)
I20260620 09:26:31.406090 23762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:31.403241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.131:39089 (local address 127.16.43.130:35995)
0620 09:26:31.403336 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:31.403348 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:31.403392 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:31.403566 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:31.403567 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:31.403569 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:26:31.403645 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:31.403647 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:31.404431 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:31.404432 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:31.405269 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:31.405271 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:31.405831 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:31.405832 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:31.405912 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:31.405916 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 09:26:31.405975 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 09:26:31.406012 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76,"mutex_wait_us":49}
I20260620 09:26:31.406101 23918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:31.403346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.130:35995 (local address 127.16.43.131:39089)
0620 09:26:31.403389 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:31.403392 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:31.403401 (+ 9us) server_negotiation.cc:407] Connection header received
0620 09:26:31.403451 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:31.403452 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:31.403462 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:31.403521 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:26:31.403779 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:31.404351 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:31.405421 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:31.405760 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:31.405816 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:31.405823 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:31.406015 (+ 192us) server_negotiation.cc:299] Negotiation successful
0620 09:26:31.406052 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 09:26:31.406466 23846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9641b40654a4c1199149f3c742b8595" candidate_uuid: "318c1dc31882444aa5becb0236f255ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a91f37200fc4719b1d4853f3fbaefe3" is_pre_election: true
I20260620 09:26:31.406634 23919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:31.403424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.130:56223 (local address 127.16.43.129:37091)
0620 09:26:31.403487 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:31.403489 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:31.403503 (+ 14us) server_negotiation.cc:407] Connection header received
0620 09:26:31.403644 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:31.403645 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:31.403656 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:31.403718 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:26:31.403932 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:31.404380 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:31.405374 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:31.406283 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:31.406354 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:31.406362 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:31.406537 (+ 175us) server_negotiation.cc:299] Negotiation successful
0620 09:26:31.406576 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 09:26:31.406651 23846 raft_consensus.cc:2454] T f9641b40654a4c1199149f3c742b8595 P 3a91f37200fc4719b1d4853f3fbaefe3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 318c1dc31882444aa5becb0236f255ea in term 0.
I20260620 09:26:31.406657 23761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:31.403276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.129:37091 (local address 127.16.43.130:56223)
0620 09:26:31.403462 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:31.403472 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:31.403511 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:31.403794 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:31.403795 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:31.403796 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:26:31.403853 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:31.403854 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:31.404487 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:31.404492 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:31.405275 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:31.405277 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:31.406369 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:31.406370 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:31.406452 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:31.406455 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 09:26:31.406517 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 09:26:31.406557 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"mutex_wait_us":18}
I20260620 09:26:31.406924 23645 leader_election.cc:302] T f9641b40654a4c1199149f3c742b8595 P 318c1dc31882444aa5becb0236f255ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 318c1dc31882444aa5becb0236f255ea, 3a91f37200fc4719b1d4853f3fbaefe3; no voters:
I20260620 09:26:31.406986 23573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9641b40654a4c1199149f3c742b8595" candidate_uuid: "318c1dc31882444aa5becb0236f255ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "625889083e23420fb414432cf88b0b51" is_pre_election: true
I20260620 09:26:31.407069 23928 raft_consensus.cc:2781] T f9641b40654a4c1199149f3c742b8595 P 318c1dc31882444aa5becb0236f255ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 09:26:31.407131 23928 raft_consensus.cc:482] T f9641b40654a4c1199149f3c742b8595 P 318c1dc31882444aa5becb0236f255ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 09:26:31.407138 23573 raft_consensus.cc:2454] T f9641b40654a4c1199149f3c742b8595 P 625889083e23420fb414432cf88b0b51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 318c1dc31882444aa5becb0236f255ea in term 0.
I20260620 09:26:31.407157 23928 raft_consensus.cc:3037] T f9641b40654a4c1199149f3c742b8595 P 318c1dc31882444aa5becb0236f255ea [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:26:31.407979 23928 raft_consensus.cc:504] T f9641b40654a4c1199149f3c742b8595 P 318c1dc31882444aa5becb0236f255ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a91f37200fc4719b1d4853f3fbaefe3" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 39089 } } peers { permanent_uuid: "318c1dc31882444aa5becb0236f255ea" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 38265 } } peers { permanent_uuid: "625889083e23420fb414432cf88b0b51" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 37091 } }
I20260620 09:26:31.408118 23928 leader_election.cc:288] T f9641b40654a4c1199149f3c742b8595 P 318c1dc31882444aa5becb0236f255ea [CANDIDATE]: Term 1 election: Requested vote from peers 3a91f37200fc4719b1d4853f3fbaefe3 (127.16.43.131:39089), 625889083e23420fb414432cf88b0b51 (127.16.43.129:37091)
I20260620 09:26:31.408303 23573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9641b40654a4c1199149f3c742b8595" candidate_uuid: "318c1dc31882444aa5becb0236f255ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "625889083e23420fb414432cf88b0b51"
I20260620 09:26:31.408318 23846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9641b40654a4c1199149f3c742b8595" candidate_uuid: "318c1dc31882444aa5becb0236f255ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a91f37200fc4719b1d4853f3fbaefe3"
I20260620 09:26:31.408393 23573 raft_consensus.cc:3037] T f9641b40654a4c1199149f3c742b8595 P 625889083e23420fb414432cf88b0b51 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:26:31.408392 23846 raft_consensus.cc:3037] T f9641b40654a4c1199149f3c742b8595 P 3a91f37200fc4719b1d4853f3fbaefe3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:26:31.409319 23573 raft_consensus.cc:2454] T f9641b40654a4c1199149f3c742b8595 P 625889083e23420fb414432cf88b0b51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 318c1dc31882444aa5becb0236f255ea in term 1.
I20260620 09:26:31.409507 23646 leader_election.cc:302] T f9641b40654a4c1199149f3c742b8595 P 318c1dc31882444aa5becb0236f255ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 318c1dc31882444aa5becb0236f255ea, 625889083e23420fb414432cf88b0b51; no voters:
I20260620 09:26:31.409605 23928 raft_consensus.cc:2781] T f9641b40654a4c1199149f3c742b8595 P 318c1dc31882444aa5becb0236f255ea [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:26:31.409687 23846 raft_consensus.cc:2454] T f9641b40654a4c1199149f3c742b8595 P 3a91f37200fc4719b1d4853f3fbaefe3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 318c1dc31882444aa5becb0236f255ea in term 1.
I20260620 09:26:31.409772 23928 raft_consensus.cc:686] T f9641b40654a4c1199149f3c742b8595 P 318c1dc31882444aa5becb0236f255ea [term 1 LEADER]: Becoming Leader. State: Replica: 318c1dc31882444aa5becb0236f255ea, State: Running, Role: LEADER
I20260620 09:26:31.409936 23928 consensus_queue.cc:237] T f9641b40654a4c1199149f3c742b8595 P 318c1dc31882444aa5becb0236f255ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a91f37200fc4719b1d4853f3fbaefe3" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 39089 } } peers { permanent_uuid: "318c1dc31882444aa5becb0236f255ea" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 38265 } } peers { permanent_uuid: "625889083e23420fb414432cf88b0b51" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 37091 } }
I20260620 09:26:31.410856 22657 catalog_manager.cc:5495] T f9641b40654a4c1199149f3c742b8595 P 318c1dc31882444aa5becb0236f255ea reported cstate change: term changed from 0 to 1, leader changed from <none> to 318c1dc31882444aa5becb0236f255ea (127.16.43.130). New cstate: current_term: 1 leader_uuid: "318c1dc31882444aa5becb0236f255ea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a91f37200fc4719b1d4853f3fbaefe3" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 39089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "318c1dc31882444aa5becb0236f255ea" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 38265 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "625889083e23420fb414432cf88b0b51" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 37091 } health_report { overall_health: UNKNOWN } } }
I20260620 09:26:31.425444 23920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:31.422876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33110 (local address 127.16.43.130:38265)
0620 09:26:31.422944 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:31.422946 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:31.422956 (+ 10us) server_negotiation.cc:407] Connection header received
0620 09:26:31.423034 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:31.423035 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:31.423056 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:31.423141 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:26:31.423293 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:31.423861 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:31.424614 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:31.424817 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:31.424903 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:31.425080 (+ 177us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:26:31.425201 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:26:31.425306 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:31.425326 (+ 20us) server_negotiation.cc:299] Negotiation successful
0620 09:26:31.425360 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 09:26:31.427297 23846 raft_consensus.cc:1261] T f9641b40654a4c1199149f3c742b8595 P 3a91f37200fc4719b1d4853f3fbaefe3 [term 1 FOLLOWER]: Refusing update from remote peer 318c1dc31882444aa5becb0236f255ea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:26:31.427304 23573 raft_consensus.cc:1261] T f9641b40654a4c1199149f3c742b8595 P 625889083e23420fb414432cf88b0b51 [term 1 FOLLOWER]: Refusing update from remote peer 318c1dc31882444aa5becb0236f255ea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:26:31.427708 23930 consensus_queue.cc:1035] T f9641b40654a4c1199149f3c742b8595 P 318c1dc31882444aa5becb0236f255ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a91f37200fc4719b1d4853f3fbaefe3" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 39089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:26:31.427824 23928 consensus_queue.cc:1035] T f9641b40654a4c1199149f3c742b8595 P 318c1dc31882444aa5becb0236f255ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "625889083e23420fb414432cf88b0b51" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 37091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:26:31.431748 23933 mvcc.cc:205] Tried to move back new op lower bound from 7298857334483517440 to 7298857334415781888. Current Snapshot: MvccSnapshot[applied={T|T < 7298857334483517440 or (T in {7298857334483517440})}]
I20260620 09:26:31.432945 23935 mvcc.cc:205] Tried to move back new op lower bound from 7298857334483517440 to 7298857334415781888. Current Snapshot: MvccSnapshot[applied={T|T < 7298857334483517440 or (T in {7298857334483517440})}]
I20260620 09:26:31.433490 22656 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40588:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:26:31.433583 22656 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40588:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:26:31.435091 22656 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1d2095405b6a4278a42cc71c3c787e3e: Sending DeleteTablet for 3 replicas of tablet f9641b40654a4c1199149f3c742b8595
I20260620 09:26:31.435497 23690 tablet_service.cc:1514] Processing DeleteTablet for tablet f9641b40654a4c1199149f3c742b8595 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:26:31 UTC) from {username='oryx', principal='oryx/127.16.43.190@KRBTEST.COM'} at 127.0.0.1:33106
I20260620 09:26:31.435671 23553 tablet_service.cc:1514] Processing DeleteTablet for tablet f9641b40654a4c1199149f3c742b8595 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:26:31 UTC) from {username='oryx', principal='oryx/127.16.43.190@KRBTEST.COM'} at 127.0.0.1:34000
I20260620 09:26:31.435741 23943 tablet_replica.cc:330] T f9641b40654a4c1199149f3c742b8595 P 318c1dc31882444aa5becb0236f255ea: stopping tablet replica
I20260620 09:26:31.435835 23943 raft_consensus.cc:2229] T f9641b40654a4c1199149f3c742b8595 P 318c1dc31882444aa5becb0236f255ea [term 1 LEADER]: Raft consensus shutting down.
I20260620 09:26:31.435899 23826 tablet_service.cc:1514] Processing DeleteTablet for tablet f9641b40654a4c1199149f3c742b8595 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:26:31 UTC) from {username='oryx', principal='oryx/127.16.43.190@KRBTEST.COM'} at 127.0.0.1:59888
I20260620 09:26:31.435941 23943 raft_consensus.cc:2258] T f9641b40654a4c1199149f3c742b8595 P 318c1dc31882444aa5becb0236f255ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:26:31.436117 23945 tablet_replica.cc:330] T f9641b40654a4c1199149f3c742b8595 P 3a91f37200fc4719b1d4853f3fbaefe3: stopping tablet replica
I20260620 09:26:31.436210 23945 raft_consensus.cc:2229] T f9641b40654a4c1199149f3c742b8595 P 3a91f37200fc4719b1d4853f3fbaefe3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:26:31.436316 23945 raft_consensus.cc:2258] T f9641b40654a4c1199149f3c742b8595 P 3a91f37200fc4719b1d4853f3fbaefe3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:26:31.436726 23945 ts_tablet_manager.cc:1905] T f9641b40654a4c1199149f3c742b8595 P 3a91f37200fc4719b1d4853f3fbaefe3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:26:31.436923 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 22625
I20260620 09:26:31.439961 23945 ts_tablet_manager.cc:1918] T f9641b40654a4c1199149f3c742b8595 P 3a91f37200fc4719b1d4853f3fbaefe3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:26:31.440043 23945 log.cc:1196] T f9641b40654a4c1199149f3c742b8595 P 3a91f37200fc4719b1d4853f3fbaefe3: Deleting WAL directory at /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/ts-2/wal/wals/f9641b40654a4c1199149f3c742b8595
I20260620 09:26:31.440313 23945 ts_tablet_manager.cc:1939] T f9641b40654a4c1199149f3c742b8595 P 3a91f37200fc4719b1d4853f3fbaefe3: Deleting consensus metadata
W20260620 09:26:31.442937 23779 connection.cc:535] server connection from 127.0.0.1:59888 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 09:26:31.443718 23943 ts_tablet_manager.cc:1905] T f9641b40654a4c1199149f3c742b8595 P 318c1dc31882444aa5becb0236f255ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:26:31.445062 23623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:31.436791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36208 (local address 127.16.43.188:46141)
0620 09:26:31.436832 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:31.436835 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:31.436842 (+ 7us) server_negotiation.cc:407] Connection header received
0620 09:26:31.436862 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:31.436863 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:31.436873 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:31.436918 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:31.437729 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:31.438470 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:31.439563 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:31.439891 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:31.439936 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:31.441833 (+ 1897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:31.441835 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:31.441836 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:26:31.441844 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:31.443689 (+ 1845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:31.444170 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:31.444171 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:31.444171 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:31.444240 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:31.444456 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:31.444457 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:31.444457 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:31.444589 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:31.444660 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:31.444880 (+ 220us) server_negotiation.cc:299] Negotiation successful
0620 09:26:31.444962 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 09:26:31.445673 23943 ts_tablet_manager.cc:1918] T f9641b40654a4c1199149f3c742b8595 P 318c1dc31882444aa5becb0236f255ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:26:31.445708 23944 tablet_replica.cc:330] T f9641b40654a4c1199149f3c742b8595 P 625889083e23420fb414432cf88b0b51: stopping tablet replica
I20260620 09:26:31.445757 23943 log.cc:1196] T f9641b40654a4c1199149f3c742b8595 P 318c1dc31882444aa5becb0236f255ea: Deleting WAL directory at /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/ts-1/wal/wals/f9641b40654a4c1199149f3c742b8595
I20260620 09:26:31.445837 23944 raft_consensus.cc:2229] T f9641b40654a4c1199149f3c742b8595 P 625889083e23420fb414432cf88b0b51 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:26:31.445964 23944 raft_consensus.cc:2258] T f9641b40654a4c1199149f3c742b8595 P 625889083e23420fb414432cf88b0b51 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:26:31.446038 23943 ts_tablet_manager.cc:1939] T f9641b40654a4c1199149f3c742b8595 P 318c1dc31882444aa5becb0236f255ea: Deleting consensus metadata
W20260620 09:26:31.446417 23644 connection.cc:441] server connection from 127.0.0.1:33106 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33106 (request call id 1) could send its response
I20260620 09:26:31.446455 23944 ts_tablet_manager.cc:1905] T f9641b40654a4c1199149f3c742b8595 P 625889083e23420fb414432cf88b0b51: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:26:31.447956 23944 ts_tablet_manager.cc:1918] T f9641b40654a4c1199149f3c742b8595 P 625889083e23420fb414432cf88b0b51: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:26:31.448027 23944 log.cc:1196] T f9641b40654a4c1199149f3c742b8595 P 625889083e23420fb414432cf88b0b51: Deleting WAL directory at /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/ts-0/wal/wals/f9641b40654a4c1199149f3c742b8595
I20260620 09:26:31.448267 23944 ts_tablet_manager.cc:1939] T f9641b40654a4c1199149f3c742b8595 P 625889083e23420fb414432cf88b0b51: Deleting consensus metadata
W20260620 09:26:31.448596 23509 connection.cc:441] server connection from 127.0.0.1:34000 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34000 (request call id 1) could send its response
I20260620 09:26:31.453737 23305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:31.444762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42472 (local address 127.16.43.189:33891)
0620 09:26:31.444809 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:31.444812 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:31.445445 (+ 633us) server_negotiation.cc:407] Connection header received
0620 09:26:31.445546 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:31.445547 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:31.445559 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:31.445591 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:31.446739 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:31.447586 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:31.448323 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:31.448637 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:31.448703 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:31.450676 (+ 1973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:31.450677 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:31.450678 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:26:31.450685 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:31.452392 (+ 1707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:31.452783 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:31.452785 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:31.452815 (+ 30us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:31.452880 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:31.453128 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:31.453129 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:31.453130 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:31.453315 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:31.453417 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:31.453520 (+ 103us) server_negotiation.cc:299] Negotiation successful
0620 09:26:31.453606 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 09:26:31.453936 23906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:31.445084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36222 (local address 127.16.43.188:46141)
0620 09:26:31.445123 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:31.445124 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:31.446862 (+ 1738us) server_negotiation.cc:407] Connection header received
0620 09:26:31.446887 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:31.446888 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:31.446901 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:31.446966 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:31.447604 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:31.448366 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:31.449293 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:31.449496 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:31.449582 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:31.450872 (+ 1290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:31.450874 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:31.450874 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:26:31.450881 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:31.452707 (+ 1826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:31.453051 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:31.453052 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:31.453053 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:31.453115 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:31.453320 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:31.453321 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:31.453321 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:31.453446 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:31.453532 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:31.453758 (+ 226us) server_negotiation.cc:299] Negotiation successful
0620 09:26:31.453840 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 09:26:31.986441 23490 catalog_manager.cc:1259] Loaded cluster ID: 76574a0d1fa54743b9e218ccdfafe710
I20260620 09:26:31.986507 23490 catalog_manager.cc:1552] T 00000000000000000000000000000000 P fc318f0d33f1422abfc6e301edf09951: loading cluster ID for follower catalog manager: success
I20260620 09:26:31.987274 23490 catalog_manager.cc:1574] T 00000000000000000000000000000000 P fc318f0d33f1422abfc6e301edf09951: acquiring CA information for follower catalog manager: success
I20260620 09:26:31.987498 23490 catalog_manager.cc:1602] T 00000000000000000000000000000000 P fc318f0d33f1422abfc6e301edf09951: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 09:26:32.432286 23954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:32.431972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:40995 (local address 127.16.43.131:38921)
0620 09:26:32.432132 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:32.432224 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:40995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":79,"threads_started":1}
W20260620 09:26:32.432569 23893 heartbeater.cc:644] Failed to heartbeat to 127.16.43.190:40995 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.16.43.190:40995: connect: Connection refused (error 111)
I20260620 09:26:32.433280 23955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:32.433016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:40995 (local address 127.16.43.130:35269)
0620 09:26:32.433139 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:32.433211 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:40995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":46,"threads_started":1}
W20260620 09:26:32.433632 23757 heartbeater.cc:644] Failed to heartbeat to 127.16.43.190:40995 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.16.43.190:40995: connect: Connection refused (error 111)
I20260620 09:26:32.433673 23956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:32.433411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:40995 (local address 127.16.43.129:48247)
0620 09:26:32.433568 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:32.433627 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:40995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":48,"threads_started":1}
W20260620 09:26:32.433862 23620 heartbeater.cc:644] Failed to heartbeat to 127.16.43.190:40995 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.16.43.190:40995: connect: Connection refused (error 111)
I20260620 09:26:32.935441 23957 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc318f0d33f1422abfc6e301edf09951 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1d2095405b6a4278a42cc71c3c787e3e)
I20260620 09:26:32.935667 23957 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc318f0d33f1422abfc6e301edf09951 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1d2095405b6a4278a42cc71c3c787e3e" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 40995 } } peers { permanent_uuid: "13835b763e164a3f9232bac524dd0bac" member_type: VOTER last_known_addr { host: "127.16.43.189" port: 33891 } attrs { promote: false } } peers { permanent_uuid: "fc318f0d33f1422abfc6e301edf09951" member_type: VOTER last_known_addr { host: "127.16.43.188" port: 46141 } attrs { promote: false } }
I20260620 09:26:32.935921 23958 raft_consensus.cc:482] T 00000000000000000000000000000000 P 13835b763e164a3f9232bac524dd0bac [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1d2095405b6a4278a42cc71c3c787e3e)
I20260620 09:26:32.936013 23957 leader_election.cc:288] T 00000000000000000000000000000000 P fc318f0d33f1422abfc6e301edf09951 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1d2095405b6a4278a42cc71c3c787e3e (127.16.43.190:40995), 13835b763e164a3f9232bac524dd0bac (127.16.43.189:33891)
I20260620 09:26:32.936060 23958 raft_consensus.cc:504] T 00000000000000000000000000000000 P 13835b763e164a3f9232bac524dd0bac [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1d2095405b6a4278a42cc71c3c787e3e" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 40995 } } peers { permanent_uuid: "13835b763e164a3f9232bac524dd0bac" member_type: VOTER last_known_addr { host: "127.16.43.189" port: 33891 } attrs { promote: false } } peers { permanent_uuid: "fc318f0d33f1422abfc6e301edf09951" member_type: VOTER last_known_addr { host: "127.16.43.188" port: 46141 } attrs { promote: false } }
I20260620 09:26:32.936349 23958 leader_election.cc:288] T 00000000000000000000000000000000 P 13835b763e164a3f9232bac524dd0bac [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1d2095405b6a4278a42cc71c3c787e3e (127.16.43.190:40995), fc318f0d33f1422abfc6e301edf09951 (127.16.43.188:46141)
I20260620 09:26:32.936523 23959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:32.936180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:40995 (local address 127.0.0.1:41034)
0620 09:26:32.936305 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:32.936388 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:40995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":90,"threads_started":1}
I20260620 09:26:32.936753 23961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:32.936468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:40995 (local address 127.0.0.1:41038)
0620 09:26:32.936642 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:32.936693 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:40995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":48,"threads_started":1}
W20260620 09:26:32.936743 23426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.43.190:40995: connect: Connection refused (error 111)
W20260620 09:26:32.936952 23240 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.43.190:40995: connect: Connection refused (error 111)
I20260620 09:26:32.937304 23959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:32.937152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:40995 (local address 127.0.0.1:41052)
0620 09:26:32.937247 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:32.937273 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:40995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84,"thread_start_us":56,"threads_started":1}
I20260620 09:26:32.937346 23964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:32.937256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:40995 (local address 127.0.0.1:41068)
0620 09:26:32.937291 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:32.937319 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:40995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260620 09:26:32.937423 23240 leader_election.cc:334] T 00000000000000000000000000000000 P 13835b763e164a3f9232bac524dd0bac [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1d2095405b6a4278a42cc71c3c787e3e (127.16.43.190:40995): Network error: Client connection negotiation failed: client connection to 127.16.43.190:40995: connect: Connection refused (error 111)
W20260620 09:26:32.937537 23426 leader_election.cc:334] T 00000000000000000000000000000000 P fc318f0d33f1422abfc6e301edf09951 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1d2095405b6a4278a42cc71c3c787e3e (127.16.43.190:40995): Network error: Client connection negotiation failed: client connection to 127.16.43.190:40995: connect: Connection refused (error 111)
I20260620 09:26:32.940151 23960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:32.936391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.189:33891 (local address 127.0.0.1:42476)
0620 09:26:32.936488 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:32.936498 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:32.936572 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:32.936805 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:32.936806 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:32.936808 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:26:32.936871 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:32.936873 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:32.937874 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:32.937875 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:32.939074 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:32.939077 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:32.939868 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:32.939869 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:32.939917 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:32.939933 (+ 16us) client_negotiation.cc:769] Sending connection context
0620 09:26:32.940015 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 09:26:32.940064 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83,"thread_start_us":48,"threads_started":1}
I20260620 09:26:32.940176 23962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:32.936493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42476 (local address 127.16.43.189:33891)
0620 09:26:32.936585 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:32.936587 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:32.936598 (+ 11us) server_negotiation.cc:407] Connection header received
0620 09:26:32.936671 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:32.936672 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:32.936683 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:32.936750 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:26:32.936949 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:32.937789 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:32.939170 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:32.939773 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:32.939848 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:32.939860 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:32.940036 (+ 176us) server_negotiation.cc:299] Negotiation successful
0620 09:26:32.940091 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76,"thread_start_us":47,"threads_started":1}
I20260620 09:26:32.940296 23961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:32.936796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.188:46141 (local address 127.0.0.1:36224)
0620 09:26:32.936809 (+ 13us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:32.936819 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:32.936864 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:32.937202 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:32.937203 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:32.937205 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:26:32.937241 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:32.937242 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:32.938085 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:32.938086 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:32.939215 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:32.939218 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:32.939991 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:32.939992 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:32.940066 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:32.940071 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 09:26:32.940124 (+ 53us) client_negotiation.cc:240] Negotiation successful
0620 09:26:32.940167 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5,"thread_start_us":82,"threads_started":1}
I20260620 09:26:32.940253 23963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:32.936724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36224 (local address 127.16.43.188:46141)
0620 09:26:32.937074 (+ 350us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:32.937076 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:32.937083 (+ 7us) server_negotiation.cc:407] Connection header received
0620 09:26:32.937101 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:32.937102 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:32.937110 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:32.937298 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:26:32.937326 (+ 28us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:32.938006 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:32.939307 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:32.939918 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:32.939966 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:32.940048 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:32.940140 (+ 92us) server_negotiation.cc:299] Negotiation successful
0620 09:26:32.940183 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":39,"threads_started":1}
I20260620 09:26:32.940528 23450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "13835b763e164a3f9232bac524dd0bac" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "fc318f0d33f1422abfc6e301edf09951" is_pre_election: true
I20260620 09:26:32.940661 23450 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fc318f0d33f1422abfc6e301edf09951 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13835b763e164a3f9232bac524dd0bac in term 1.
I20260620 09:26:32.940661 23264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fc318f0d33f1422abfc6e301edf09951" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "13835b763e164a3f9232bac524dd0bac" is_pre_election: true
I20260620 09:26:32.940760 23264 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 13835b763e164a3f9232bac524dd0bac [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc318f0d33f1422abfc6e301edf09951 in term 1.
I20260620 09:26:32.940821 23240 leader_election.cc:302] T 00000000000000000000000000000000 P 13835b763e164a3f9232bac524dd0bac [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 13835b763e164a3f9232bac524dd0bac, fc318f0d33f1422abfc6e301edf09951; no voters: 1d2095405b6a4278a42cc71c3c787e3e
I20260620 09:26:32.940897 23426 leader_election.cc:302] T 00000000000000000000000000000000 P fc318f0d33f1422abfc6e301edf09951 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 13835b763e164a3f9232bac524dd0bac, fc318f0d33f1422abfc6e301edf09951; no voters: 1d2095405b6a4278a42cc71c3c787e3e
I20260620 09:26:32.940928 23958 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 13835b763e164a3f9232bac524dd0bac [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 09:26:32.940975 23958 raft_consensus.cc:482] T 00000000000000000000000000000000 P 13835b763e164a3f9232bac524dd0bac [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1d2095405b6a4278a42cc71c3c787e3e)
I20260620 09:26:32.941005 23958 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 13835b763e164a3f9232bac524dd0bac [term 1 FOLLOWER]: Advancing to term 2
I20260620 09:26:32.941010 23957 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc318f0d33f1422abfc6e301edf09951 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 09:26:32.941038 23957 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc318f0d33f1422abfc6e301edf09951 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1d2095405b6a4278a42cc71c3c787e3e)
I20260620 09:26:32.941071 23957 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc318f0d33f1422abfc6e301edf09951 [term 1 FOLLOWER]: Advancing to term 2
I20260620 09:26:32.941738 23958 raft_consensus.cc:504] T 00000000000000000000000000000000 P 13835b763e164a3f9232bac524dd0bac [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1d2095405b6a4278a42cc71c3c787e3e" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 40995 } } peers { permanent_uuid: "13835b763e164a3f9232bac524dd0bac" member_type: VOTER last_known_addr { host: "127.16.43.189" port: 33891 } attrs { promote: false } } peers { permanent_uuid: "fc318f0d33f1422abfc6e301edf09951" member_type: VOTER last_known_addr { host: "127.16.43.188" port: 46141 } attrs { promote: false } }
I20260620 09:26:32.941748 23957 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc318f0d33f1422abfc6e301edf09951 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1d2095405b6a4278a42cc71c3c787e3e" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 40995 } } peers { permanent_uuid: "13835b763e164a3f9232bac524dd0bac" member_type: VOTER last_known_addr { host: "127.16.43.189" port: 33891 } attrs { promote: false } } peers { permanent_uuid: "fc318f0d33f1422abfc6e301edf09951" member_type: VOTER last_known_addr { host: "127.16.43.188" port: 46141 } attrs { promote: false } }
I20260620 09:26:32.941880 23958 leader_election.cc:288] T 00000000000000000000000000000000 P 13835b763e164a3f9232bac524dd0bac [CANDIDATE]: Term 2 election: Requested vote from peers 1d2095405b6a4278a42cc71c3c787e3e (127.16.43.190:40995), fc318f0d33f1422abfc6e301edf09951 (127.16.43.188:46141)
I20260620 09:26:32.941880 23957 leader_election.cc:288] T 00000000000000000000000000000000 P fc318f0d33f1422abfc6e301edf09951 [CANDIDATE]: Term 2 election: Requested vote from peers 1d2095405b6a4278a42cc71c3c787e3e (127.16.43.190:40995), 13835b763e164a3f9232bac524dd0bac (127.16.43.189:33891)
I20260620 09:26:32.942090 23960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:32.941970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:40995 (local address 127.0.0.1:41078)
0620 09:26:32.942032 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:32.942060 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:40995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260620 09:26:32.942132 23961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:32.942023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:40995 (local address 127.0.0.1:41088)
0620 09:26:32.942079 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:32.942107 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:40995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260620 09:26:32.942142 23264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fc318f0d33f1422abfc6e301edf09951" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "13835b763e164a3f9232bac524dd0bac"
I20260620 09:26:32.942179 23450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "13835b763e164a3f9232bac524dd0bac" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "fc318f0d33f1422abfc6e301edf09951"
I20260620 09:26:32.942251 23264 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 13835b763e164a3f9232bac524dd0bac [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate fc318f0d33f1422abfc6e301edf09951 in current term 2: Already voted for candidate 13835b763e164a3f9232bac524dd0bac in this term.
I20260620 09:26:32.942265 23450 raft_consensus.cc:2379] T 00000000000000000000000000000000 P fc318f0d33f1422abfc6e301edf09951 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 13835b763e164a3f9232bac524dd0bac in current term 2: Already voted for candidate fc318f0d33f1422abfc6e301edf09951 in this term.
I20260620 09:26:32.942456 23960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:32.942302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:40995 (local address 127.0.0.1:41096)
0620 09:26:32.942407 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:32.942441 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:40995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
W20260620 09:26:32.942548 23426 leader_election.cc:334] T 00000000000000000000000000000000 P fc318f0d33f1422abfc6e301edf09951 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 1d2095405b6a4278a42cc71c3c787e3e (127.16.43.190:40995): Network error: Client connection negotiation failed: client connection to 127.16.43.190:40995: connect: Connection refused (error 111)
I20260620 09:26:32.942616 23961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:32.942510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:40995 (local address 127.0.0.1:41110)
0620 09:26:32.942564 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:32.942589 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:40995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260620 09:26:32.942633 23426 leader_election.cc:302] T 00000000000000000000000000000000 P fc318f0d33f1422abfc6e301edf09951 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fc318f0d33f1422abfc6e301edf09951; no voters: 13835b763e164a3f9232bac524dd0bac, 1d2095405b6a4278a42cc71c3c787e3e
W20260620 09:26:32.942677 23240 leader_election.cc:334] T 00000000000000000000000000000000 P 13835b763e164a3f9232bac524dd0bac [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 1d2095405b6a4278a42cc71c3c787e3e (127.16.43.190:40995): Network error: Client connection negotiation failed: client connection to 127.16.43.190:40995: connect: Connection refused (error 111)
I20260620 09:26:32.942741 23957 raft_consensus.cc:2726] T 00000000000000000000000000000000 P fc318f0d33f1422abfc6e301edf09951 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260620 09:26:32.942744 23240 leader_election.cc:302] T 00000000000000000000000000000000 P 13835b763e164a3f9232bac524dd0bac [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 13835b763e164a3f9232bac524dd0bac; no voters: 1d2095405b6a4278a42cc71c3c787e3e, fc318f0d33f1422abfc6e301edf09951
I20260620 09:26:32.942826 23958 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 13835b763e164a3f9232bac524dd0bac [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260620 09:26:33.433341 23968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:33.432958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:40995 (local address 127.16.43.131:59731)
0620 09:26:33.433197 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:33.433265 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:40995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":96,"threads_started":1}
I20260620 09:26:33.434451 23969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:33.434065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:40995 (local address 127.16.43.130:59523)
0620 09:26:33.434340 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:33.434387 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:40995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":66,"threads_started":1}
I20260620 09:26:33.434609 23970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:33.434391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:40995 (local address 127.16.43.129:33517)
0620 09:26:33.434510 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:33.434548 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:40995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":73,"threads_started":1}
I20260620 09:26:34.434106 23971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:34.433843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:40995 (local address 127.16.43.131:55687)
0620 09:26:34.433998 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:34.434044 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:40995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":106,"threads_started":1}
I20260620 09:26:34.435302 23972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:34.435077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:40995 (local address 127.16.43.129:59855)
0620 09:26:34.435211 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:34.435246 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:40995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":55,"threads_started":1}
I20260620 09:26:34.435365 23973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:34.435155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:40995 (local address 127.16.43.130:44589)
0620 09:26:34.435274 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:34.435314 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:40995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":74,"threads_started":1}
I20260620 09:26:34.799319 23974 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc318f0d33f1422abfc6e301edf09951 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:26:34.799427 23974 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc318f0d33f1422abfc6e301edf09951 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1d2095405b6a4278a42cc71c3c787e3e" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 40995 } } peers { permanent_uuid: "13835b763e164a3f9232bac524dd0bac" member_type: VOTER last_known_addr { host: "127.16.43.189" port: 33891 } attrs { promote: false } } peers { permanent_uuid: "fc318f0d33f1422abfc6e301edf09951" member_type: VOTER last_known_addr { host: "127.16.43.188" port: 46141 } attrs { promote: false } }
I20260620 09:26:34.799670 23974 leader_election.cc:288] T 00000000000000000000000000000000 P fc318f0d33f1422abfc6e301edf09951 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1d2095405b6a4278a42cc71c3c787e3e (127.16.43.190:40995), 13835b763e164a3f9232bac524dd0bac (127.16.43.189:33891)
I20260620 09:26:34.799997 23975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:34.799808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:40995 (local address 127.0.0.1:41368)
0620 09:26:34.799906 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:34.799957 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:40995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75,"thread_start_us":44,"threads_started":1}
I20260620 09:26:34.800086 23264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fc318f0d33f1422abfc6e301edf09951" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "13835b763e164a3f9232bac524dd0bac" is_pre_election: true
I20260620 09:26:34.800195 23264 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 13835b763e164a3f9232bac524dd0bac [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc318f0d33f1422abfc6e301edf09951 in term 2.
I20260620 09:26:34.800385 23426 leader_election.cc:302] T 00000000000000000000000000000000 P fc318f0d33f1422abfc6e301edf09951 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13835b763e164a3f9232bac524dd0bac, fc318f0d33f1422abfc6e301edf09951; no voters:
I20260620 09:26:34.800450 23975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:34.800337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:40995 (local address 127.0.0.1:41382)
0620 09:26:34.800395 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:34.800429 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:40995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260620 09:26:34.800508 23974 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc318f0d33f1422abfc6e301edf09951 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260620 09:26:34.800535 23426 leader_election.cc:334] T 00000000000000000000000000000000 P fc318f0d33f1422abfc6e301edf09951 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1d2095405b6a4278a42cc71c3c787e3e (127.16.43.190:40995): Network error: Client connection negotiation failed: client connection to 127.16.43.190:40995: connect: Connection refused (error 111)
I20260620 09:26:34.800546 23974 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc318f0d33f1422abfc6e301edf09951 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 09:26:34.800603 23974 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc318f0d33f1422abfc6e301edf09951 [term 2 FOLLOWER]: Advancing to term 3
I20260620 09:26:34.801359 23974 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc318f0d33f1422abfc6e301edf09951 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1d2095405b6a4278a42cc71c3c787e3e" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 40995 } } peers { permanent_uuid: "13835b763e164a3f9232bac524dd0bac" member_type: VOTER last_known_addr { host: "127.16.43.189" port: 33891 } attrs { promote: false } } peers { permanent_uuid: "fc318f0d33f1422abfc6e301edf09951" member_type: VOTER last_known_addr { host: "127.16.43.188" port: 46141 } attrs { promote: false } }
I20260620 09:26:34.801458 23974 leader_election.cc:288] T 00000000000000000000000000000000 P fc318f0d33f1422abfc6e301edf09951 [CANDIDATE]: Term 3 election: Requested vote from peers 1d2095405b6a4278a42cc71c3c787e3e (127.16.43.190:40995), 13835b763e164a3f9232bac524dd0bac (127.16.43.189:33891)
I20260620 09:26:34.801729 23975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:34.801585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:40995 (local address 127.0.0.1:41396)
0620 09:26:34.801667 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:34.801700 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:40995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260620 09:26:34.801784 23264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fc318f0d33f1422abfc6e301edf09951" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "13835b763e164a3f9232bac524dd0bac"
I20260620 09:26:34.801877 23264 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 13835b763e164a3f9232bac524dd0bac [term 2 FOLLOWER]: Advancing to term 3
I20260620 09:26:34.802078 23975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:34.801976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:40995 (local address 127.0.0.1:41400)
0620 09:26:34.802030 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:34.802057 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:40995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260620 09:26:34.802161 23426 leader_election.cc:334] T 00000000000000000000000000000000 P fc318f0d33f1422abfc6e301edf09951 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 1d2095405b6a4278a42cc71c3c787e3e (127.16.43.190:40995): Network error: Client connection negotiation failed: client connection to 127.16.43.190:40995: connect: Connection refused (error 111)
I20260620 09:26:34.802624 23264 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 13835b763e164a3f9232bac524dd0bac [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc318f0d33f1422abfc6e301edf09951 in term 3.
I20260620 09:26:34.802886 23426 leader_election.cc:302] T 00000000000000000000000000000000 P fc318f0d33f1422abfc6e301edf09951 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 13835b763e164a3f9232bac524dd0bac, fc318f0d33f1422abfc6e301edf09951; no voters: 1d2095405b6a4278a42cc71c3c787e3e
I20260620 09:26:34.802985 23974 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc318f0d33f1422abfc6e301edf09951 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 09:26:34.803143 23974 raft_consensus.cc:686] T 00000000000000000000000000000000 P fc318f0d33f1422abfc6e301edf09951 [term 3 LEADER]: Becoming Leader. State: Replica: fc318f0d33f1422abfc6e301edf09951, State: Running, Role: LEADER
I20260620 09:26:34.803251 23974 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc318f0d33f1422abfc6e301edf09951 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1d2095405b6a4278a42cc71c3c787e3e" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 40995 } } peers { permanent_uuid: "13835b763e164a3f9232bac524dd0bac" member_type: VOTER last_known_addr { host: "127.16.43.189" port: 33891 } attrs { promote: false } } peers { permanent_uuid: "fc318f0d33f1422abfc6e301edf09951" member_type: VOTER last_known_addr { host: "127.16.43.188" port: 46141 } attrs { promote: false } }
I20260620 09:26:34.803812 23977 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc318f0d33f1422abfc6e301edf09951 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc318f0d33f1422abfc6e301edf09951. Latest consensus state: current_term: 3 leader_uuid: "fc318f0d33f1422abfc6e301edf09951" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1d2095405b6a4278a42cc71c3c787e3e" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 40995 } } peers { permanent_uuid: "13835b763e164a3f9232bac524dd0bac" member_type: VOTER last_known_addr { host: "127.16.43.189" port: 33891 } attrs { promote: false } } peers { permanent_uuid: "fc318f0d33f1422abfc6e301edf09951" member_type: VOTER last_known_addr { host: "127.16.43.188" port: 46141 } attrs { promote: false } } }
I20260620 09:26:34.803920 23977 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc318f0d33f1422abfc6e301edf09951 [sys.catalog]: This master's current role is: LEADER
I20260620 09:26:34.804081 23979 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:26:34.804497 23979 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:26:34.804560 23979 catalog_manager.cc:1259] Loaded cluster ID: 76574a0d1fa54743b9e218ccdfafe710
I20260620 09:26:34.804613 23979 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:26:34.804803 23979 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:26:34.804903 23979 catalog_manager.cc:5879] T 00000000000000000000000000000000 P fc318f0d33f1422abfc6e301edf09951: Loaded TSK: 0
I20260620 09:26:34.805051 23979 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 09:26:34.879212 23440 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36222:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 09:26:34.880959 23440 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 09:26:34.882083 23975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:34.881925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:40995 (local address 127.0.0.1:41418)
0620 09:26:34.881997 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:34.882045 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:40995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260620 09:26:34.882155 23264 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 13835b763e164a3f9232bac524dd0bac [term 3 FOLLOWER]: Refusing update from remote peer fc318f0d33f1422abfc6e301edf09951: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260620 09:26:34.882500 23975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:34.882334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:40995 (local address 127.0.0.1:41428)
0620 09:26:34.882399 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:34.882444 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:40995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260620 09:26:34.882448 23974 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fc318f0d33f1422abfc6e301edf09951 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13835b763e164a3f9232bac524dd0bac" member_type: VOTER last_known_addr { host: "127.16.43.189" port: 33891 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260620 09:26:34.882591 23426 consensus_peers.cc:487] T 00000000000000000000000000000000 P fc318f0d33f1422abfc6e301edf09951 -> Peer 1d2095405b6a4278a42cc71c3c787e3e (127.16.43.190:40995): Couldn't send request to peer 1d2095405b6a4278a42cc71c3c787e3e. Status: Network error: Client connection negotiation failed: client connection to 127.16.43.190:40995: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 09:26:34.883105 23981 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13835b763e164a3f9232bac524dd0bac [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc318f0d33f1422abfc6e301edf09951. Latest consensus state: current_term: 3 leader_uuid: "fc318f0d33f1422abfc6e301edf09951" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1d2095405b6a4278a42cc71c3c787e3e" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 40995 } } peers { permanent_uuid: "13835b763e164a3f9232bac524dd0bac" member_type: VOTER last_known_addr { host: "127.16.43.189" port: 33891 } attrs { promote: false } } peers { permanent_uuid: "fc318f0d33f1422abfc6e301edf09951" member_type: VOTER last_known_addr { host: "127.16.43.188" port: 46141 } attrs { promote: false } } }
I20260620 09:26:34.883200 23981 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13835b763e164a3f9232bac524dd0bac [sys.catalog]: This master's current role is: FOLLOWER
I20260620 09:26:34.883683 23974 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc318f0d33f1422abfc6e301edf09951 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "fc318f0d33f1422abfc6e301edf09951" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1d2095405b6a4278a42cc71c3c787e3e" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 40995 } } peers { permanent_uuid: "13835b763e164a3f9232bac524dd0bac" member_type: VOTER last_known_addr { host: "127.16.43.189" port: 33891 } attrs { promote: false } } peers { permanent_uuid: "fc318f0d33f1422abfc6e301edf09951" member_type: VOTER last_known_addr { host: "127.16.43.188" port: 46141 } attrs { promote: false } } }
I20260620 09:26:34.883771 23974 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc318f0d33f1422abfc6e301edf09951 [sys.catalog]: This master's current role is: LEADER
I20260620 09:26:34.883914 23982 mvcc.cc:205] Tried to move back new op lower bound from 7298857348634796032 to 7298857348314787840. Current Snapshot: MvccSnapshot[applied={T|T < 7298857348634796032}]
I20260620 09:26:34.884054 23981 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13835b763e164a3f9232bac524dd0bac [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "fc318f0d33f1422abfc6e301edf09951" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1d2095405b6a4278a42cc71c3c787e3e" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 40995 } } peers { permanent_uuid: "13835b763e164a3f9232bac524dd0bac" member_type: VOTER last_known_addr { host: "127.16.43.189" port: 33891 } attrs { promote: false } } peers { permanent_uuid: "fc318f0d33f1422abfc6e301edf09951" member_type: VOTER last_known_addr { host: "127.16.43.188" port: 46141 } attrs { promote: false } } }
I20260620 09:26:34.884150 23981 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13835b763e164a3f9232bac524dd0bac [sys.catalog]: This master's current role is: FOLLOWER
I20260620 09:26:34.889302 23987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:34.885778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34004 (local address 127.16.43.129:37091)
0620 09:26:34.886115 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:34.886117 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:34.886126 (+ 9us) server_negotiation.cc:407] Connection header received
0620 09:26:34.886153 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:34.886153 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:34.886164 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:34.886329 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:26:34.886492 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:34.887041 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:34.888455 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:34.888933 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:34.889026 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:34.889039 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:34.889183 (+ 144us) server_negotiation.cc:299] Negotiation successful
0620 09:26:34.889222 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":70,"threads_started":1}
I20260620 09:26:34.889343 23975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:34.885702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.129:37091 (local address 127.0.0.1:34004)
0620 09:26:34.885885 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:34.885896 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:34.885951 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:34.886289 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:34.886289 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:34.886290 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:26:34.886407 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:34.886409 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:34.887126 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:34.887127 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:34.888358 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:34.888360 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:34.889027 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:34.889028 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:34.889090 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:34.889094 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 09:26:34.889175 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 09:26:34.889221 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"mutex_wait_us":108}
I20260620 09:26:34.889516 23989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:34.885823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59890 (local address 127.16.43.131:39089)
0620 09:26:34.886046 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:34.886048 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:34.886059 (+ 11us) server_negotiation.cc:407] Connection header received
0620 09:26:34.886156 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:34.886157 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:34.886167 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:34.886234 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:26:34.886441 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:34.887246 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:34.888419 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:34.888987 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:34.889055 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:34.889066 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:34.889381 (+ 315us) server_negotiation.cc:299] Negotiation successful
0620 09:26:34.889436 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":46,"threads_started":1}
I20260620 09:26:34.889628 23553 tablet_service.cc:1467] Processing CreateTablet for tablet cb17be8a0de345f3a3be1971f6c729e8 (DEFAULT_TABLE table=test-table [id=8fda61ba944c407b98a6a9e183971295]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:26:34.889747 23553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb17be8a0de345f3a3be1971f6c729e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:26:34.889998 23990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:34.885909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33114 (local address 127.16.43.130:38265)
0620 09:26:34.886017 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:34.886019 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:34.886214 (+ 195us) server_negotiation.cc:407] Connection header received
0620 09:26:34.886283 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:34.886284 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:34.886295 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:34.886382 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:26:34.886534 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:34.887245 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:34.888535 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:34.889111 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:34.889171 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:34.889183 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:34.889888 (+ 705us) server_negotiation.cc:299] Negotiation successful
0620 09:26:34.889942 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83,"thread_start_us":53,"threads_started":1}
I20260620 09:26:34.890998 23986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:34.885809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.130:38265 (local address 127.0.0.1:33114)
0620 09:26:34.886100 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:34.886108 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:34.886154 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:34.886429 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:34.886430 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:34.886432 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:26:34.886473 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:34.886474 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:34.887328 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:34.887329 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:34.888461 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:34.888462 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:34.889211 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:34.889212 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:34.889263 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:34.889266 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 09:26:34.889309 (+ 43us) client_negotiation.cc:240] Negotiation successful
0620 09:26:34.889344 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":71,"threads_started":1}
I20260620 09:26:34.890954 23991 tablet_bootstrap.cc:492] T cb17be8a0de345f3a3be1971f6c729e8 P 625889083e23420fb414432cf88b0b51: Bootstrap starting.
I20260620 09:26:34.891101 23988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:34.885733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.131:39089 (local address 127.0.0.1:59890)
0620 09:26:34.886016 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:34.886027 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:34.886074 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:34.886282 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:34.886282 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:34.886285 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:26:34.886384 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:34.886386 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:34.887339 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:34.887340 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:34.888335 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:34.888337 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:34.889102 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:34.889103 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:34.889170 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:34.889174 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 09:26:34.889222 (+ 48us) client_negotiation.cc:240] Negotiation successful
0620 09:26:34.889264 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"mutex_wait_us":3,"thread_start_us":78,"threads_started":1}
I20260620 09:26:34.891330 23826 tablet_service.cc:1467] Processing CreateTablet for tablet cb17be8a0de345f3a3be1971f6c729e8 (DEFAULT_TABLE table=test-table [id=8fda61ba944c407b98a6a9e183971295]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:26:34.891331 23690 tablet_service.cc:1467] Processing CreateTablet for tablet cb17be8a0de345f3a3be1971f6c729e8 (DEFAULT_TABLE table=test-table [id=8fda61ba944c407b98a6a9e183971295]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:26:34.891465 23826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb17be8a0de345f3a3be1971f6c729e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:26:34.891485 23690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb17be8a0de345f3a3be1971f6c729e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:26:34.892511 23992 tablet_bootstrap.cc:492] T cb17be8a0de345f3a3be1971f6c729e8 P 3a91f37200fc4719b1d4853f3fbaefe3: Bootstrap starting.
I20260620 09:26:34.892887 23992 tablet_bootstrap.cc:654] T cb17be8a0de345f3a3be1971f6c729e8 P 3a91f37200fc4719b1d4853f3fbaefe3: Neither blocks nor log segments found. Creating new log.
I20260620 09:26:34.892989 23993 tablet_bootstrap.cc:492] T cb17be8a0de345f3a3be1971f6c729e8 P 318c1dc31882444aa5becb0236f255ea: Bootstrap starting.
I20260620 09:26:34.893203 23991 tablet_bootstrap.cc:654] T cb17be8a0de345f3a3be1971f6c729e8 P 625889083e23420fb414432cf88b0b51: Neither blocks nor log segments found. Creating new log.
I20260620 09:26:34.893419 23993 tablet_bootstrap.cc:654] T cb17be8a0de345f3a3be1971f6c729e8 P 318c1dc31882444aa5becb0236f255ea: Neither blocks nor log segments found. Creating new log.
I20260620 09:26:34.893781 23991 tablet_bootstrap.cc:492] T cb17be8a0de345f3a3be1971f6c729e8 P 625889083e23420fb414432cf88b0b51: No bootstrap required, opened a new log
I20260620 09:26:34.893842 23991 ts_tablet_manager.cc:1397] T cb17be8a0de345f3a3be1971f6c729e8 P 625889083e23420fb414432cf88b0b51: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260620 09:26:34.893909 23992 tablet_bootstrap.cc:492] T cb17be8a0de345f3a3be1971f6c729e8 P 3a91f37200fc4719b1d4853f3fbaefe3: No bootstrap required, opened a new log
I20260620 09:26:34.893959 23992 ts_tablet_manager.cc:1397] T cb17be8a0de345f3a3be1971f6c729e8 P 3a91f37200fc4719b1d4853f3fbaefe3: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 09:26:34.893988 23991 raft_consensus.cc:348] T cb17be8a0de345f3a3be1971f6c729e8 P 625889083e23420fb414432cf88b0b51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625889083e23420fb414432cf88b0b51" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 37091 } } peers { permanent_uuid: "3a91f37200fc4719b1d4853f3fbaefe3" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 39089 } } peers { permanent_uuid: "318c1dc31882444aa5becb0236f255ea" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 38265 } }
I20260620 09:26:34.894078 23991 raft_consensus.cc:374] T cb17be8a0de345f3a3be1971f6c729e8 P 625889083e23420fb414432cf88b0b51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:26:34.894099 23991 raft_consensus.cc:729] T cb17be8a0de345f3a3be1971f6c729e8 P 625889083e23420fb414432cf88b0b51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 625889083e23420fb414432cf88b0b51, State: Initialized, Role: FOLLOWER
I20260620 09:26:34.894068 23992 raft_consensus.cc:348] T cb17be8a0de345f3a3be1971f6c729e8 P 3a91f37200fc4719b1d4853f3fbaefe3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625889083e23420fb414432cf88b0b51" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 37091 } } peers { permanent_uuid: "3a91f37200fc4719b1d4853f3fbaefe3" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 39089 } } peers { permanent_uuid: "318c1dc31882444aa5becb0236f255ea" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 38265 } }
I20260620 09:26:34.894135 23992 raft_consensus.cc:374] T cb17be8a0de345f3a3be1971f6c729e8 P 3a91f37200fc4719b1d4853f3fbaefe3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:26:34.894164 23992 raft_consensus.cc:729] T cb17be8a0de345f3a3be1971f6c729e8 P 3a91f37200fc4719b1d4853f3fbaefe3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a91f37200fc4719b1d4853f3fbaefe3, State: Initialized, Role: FOLLOWER
I20260620 09:26:34.894172 23991 consensus_queue.cc:260] T cb17be8a0de345f3a3be1971f6c729e8 P 625889083e23420fb414432cf88b0b51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625889083e23420fb414432cf88b0b51" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 37091 } } peers { permanent_uuid: "3a91f37200fc4719b1d4853f3fbaefe3" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 39089 } } peers { permanent_uuid: "318c1dc31882444aa5becb0236f255ea" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 38265 } }
I20260620 09:26:34.894223 23992 consensus_queue.cc:260] T cb17be8a0de345f3a3be1971f6c729e8 P 3a91f37200fc4719b1d4853f3fbaefe3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625889083e23420fb414432cf88b0b51" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 37091 } } peers { permanent_uuid: "3a91f37200fc4719b1d4853f3fbaefe3" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 39089 } } peers { permanent_uuid: "318c1dc31882444aa5becb0236f255ea" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 38265 } }
I20260620 09:26:34.894411 23992 ts_tablet_manager.cc:1428] T cb17be8a0de345f3a3be1971f6c729e8 P 3a91f37200fc4719b1d4853f3fbaefe3: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 09:26:34.894722 23972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:34.894550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:40995 (local address 127.16.43.129:47305)
0620 09:26:34.894649 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:34.894680 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:40995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260620 09:26:34.894970 23971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:34.894671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:40995 (local address 127.16.43.131:51493)
0620 09:26:34.894914 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:34.894944 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:40995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226}
I20260620 09:26:34.895098 23892 heartbeater.cc:503] Master 127.16.43.188:46141 was elected leader, sending a full tablet report...
I20260620 09:26:34.895318 23991 ts_tablet_manager.cc:1428] T cb17be8a0de345f3a3be1971f6c729e8 P 625889083e23420fb414432cf88b0b51: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 09:26:34.895437 23619 heartbeater.cc:503] Master 127.16.43.188:46141 was elected leader, sending a full tablet report...
I20260620 09:26:34.895376 23993 tablet_bootstrap.cc:492] T cb17be8a0de345f3a3be1971f6c729e8 P 318c1dc31882444aa5becb0236f255ea: No bootstrap required, opened a new log
I20260620 09:26:34.895578 23993 ts_tablet_manager.cc:1397] T cb17be8a0de345f3a3be1971f6c729e8 P 318c1dc31882444aa5becb0236f255ea: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260620 09:26:34.895721 23972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:34.895625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:40995 (local address 127.16.43.129:58493)
0620 09:26:34.895657 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:34.895691 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:40995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260620 09:26:34.895695 23993 raft_consensus.cc:348] T cb17be8a0de345f3a3be1971f6c729e8 P 318c1dc31882444aa5becb0236f255ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625889083e23420fb414432cf88b0b51" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 37091 } } peers { permanent_uuid: "3a91f37200fc4719b1d4853f3fbaefe3" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 39089 } } peers { permanent_uuid: "318c1dc31882444aa5becb0236f255ea" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 38265 } }
I20260620 09:26:34.895758 23993 raft_consensus.cc:374] T cb17be8a0de345f3a3be1971f6c729e8 P 318c1dc31882444aa5becb0236f255ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:26:34.895779 23993 raft_consensus.cc:729] T cb17be8a0de345f3a3be1971f6c729e8 P 318c1dc31882444aa5becb0236f255ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 318c1dc31882444aa5becb0236f255ea, State: Initialized, Role: FOLLOWER
I20260620 09:26:34.895841 23993 consensus_queue.cc:260] T cb17be8a0de345f3a3be1971f6c729e8 P 318c1dc31882444aa5becb0236f255ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625889083e23420fb414432cf88b0b51" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 37091 } } peers { permanent_uuid: "3a91f37200fc4719b1d4853f3fbaefe3" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 39089 } } peers { permanent_uuid: "318c1dc31882444aa5becb0236f255ea" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 38265 } }
I20260620 09:26:34.896005 23993 ts_tablet_manager.cc:1428] T cb17be8a0de345f3a3be1971f6c729e8 P 318c1dc31882444aa5becb0236f255ea: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:34.896327 23973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:34.896187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:40995 (local address 127.16.43.130:59749)
0620 09:26:34.896259 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:34.896293 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:40995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260620 09:26:34.896680 23973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:34.896574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:40995 (local address 127.16.43.130:52161)
0620 09:26:34.896627 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:34.896653 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:40995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260620 09:26:34.896714 23756 heartbeater.cc:503] Master 127.16.43.188:46141 was elected leader, sending a full tablet report...
W20260620 09:26:34.947956 23759 tablet.cc:2369] T cb17be8a0de345f3a3be1971f6c729e8 P 318c1dc31882444aa5becb0236f255ea: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 09:26:35.045797 23622 tablet.cc:2369] T cb17be8a0de345f3a3be1971f6c729e8 P 625889083e23420fb414432cf88b0b51: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 09:26:35.099522 23895 tablet.cc:2369] T cb17be8a0de345f3a3be1971f6c729e8 P 3a91f37200fc4719b1d4853f3fbaefe3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 09:26:35.124022 23997 raft_consensus.cc:482] T cb17be8a0de345f3a3be1971f6c729e8 P 625889083e23420fb414432cf88b0b51 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:26:35.124187 23997 raft_consensus.cc:504] T cb17be8a0de345f3a3be1971f6c729e8 P 625889083e23420fb414432cf88b0b51 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625889083e23420fb414432cf88b0b51" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 37091 } } peers { permanent_uuid: "3a91f37200fc4719b1d4853f3fbaefe3" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 39089 } } peers { permanent_uuid: "318c1dc31882444aa5becb0236f255ea" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 38265 } }
I20260620 09:26:35.124598 23997 leader_election.cc:288] T cb17be8a0de345f3a3be1971f6c729e8 P 625889083e23420fb414432cf88b0b51 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3a91f37200fc4719b1d4853f3fbaefe3 (127.16.43.131:39089), 318c1dc31882444aa5becb0236f255ea (127.16.43.130:38265)
I20260620 09:26:35.127810 24000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:35.124863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.131:39089 (local address 127.16.43.129:45345)
0620 09:26:35.125070 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:35.125080 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:35.125127 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:35.125262 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:35.125263 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:35.125265 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:26:35.125339 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:35.125341 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:35.126050 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:35.126051 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:35.126898 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:35.126901 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:35.127448 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:35.127449 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:35.127520 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:35.127555 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 09:26:35.127660 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 09:26:35.127704 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"mutex_wait_us":4,"thread_start_us":142,"threads_started":1}
I20260620 09:26:35.127843 23989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:35.125070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.129:45345 (local address 127.16.43.131:39089)
0620 09:26:35.125140 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:35.125143 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:35.125152 (+ 9us) server_negotiation.cc:407] Connection header received
0620 09:26:35.125182 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:35.125183 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:35.125190 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:35.125239 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:26:35.125488 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:35.125960 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:35.127028 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:35.127371 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:35.127427 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:35.127435 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:35.127683 (+ 248us) server_negotiation.cc:299] Negotiation successful
0620 09:26:35.127728 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 09:26:35.127962 23972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:35.124823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.130:38265 (local address 127.16.43.129:47679)
0620 09:26:35.124888 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:35.124903 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:35.125031 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:35.125301 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:35.125302 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:35.125304 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:26:35.125354 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:35.125355 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:35.126070 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:35.126071 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:35.126896 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:35.126899 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:35.127613 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:35.127614 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:35.127703 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:35.127707 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 09:26:35.127764 (+ 57us) client_negotiation.cc:240] Negotiation successful
0620 09:26:35.127803 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":41}
I20260620 09:26:35.128104 23990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:35.125013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.129:47679 (local address 127.16.43.130:38265)
0620 09:26:35.125153 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:35.125156 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:35.125165 (+ 9us) server_negotiation.cc:407] Connection header received
0620 09:26:35.125195 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:35.125196 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:35.125207 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:35.125272 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:26:35.125456 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:35.125964 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:35.126992 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:35.127489 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:35.127606 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:35.127619 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:35.127785 (+ 166us) server_negotiation.cc:299] Negotiation successful
0620 09:26:35.127824 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260620 09:26:35.128156 23846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb17be8a0de345f3a3be1971f6c729e8" candidate_uuid: "625889083e23420fb414432cf88b0b51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a91f37200fc4719b1d4853f3fbaefe3" is_pre_election: true
I20260620 09:26:35.128276 23846 raft_consensus.cc:2454] T cb17be8a0de345f3a3be1971f6c729e8 P 3a91f37200fc4719b1d4853f3fbaefe3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 625889083e23420fb414432cf88b0b51 in term 0.
I20260620 09:26:35.128388 23710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb17be8a0de345f3a3be1971f6c729e8" candidate_uuid: "625889083e23420fb414432cf88b0b51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "318c1dc31882444aa5becb0236f255ea" is_pre_election: true
I20260620 09:26:35.128508 23710 raft_consensus.cc:2454] T cb17be8a0de345f3a3be1971f6c729e8 P 318c1dc31882444aa5becb0236f255ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 625889083e23420fb414432cf88b0b51 in term 0.
I20260620 09:26:35.128793 23508 leader_election.cc:302] T cb17be8a0de345f3a3be1971f6c729e8 P 625889083e23420fb414432cf88b0b51 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a91f37200fc4719b1d4853f3fbaefe3, 625889083e23420fb414432cf88b0b51; no voters:
I20260620 09:26:35.129009 23997 raft_consensus.cc:2781] T cb17be8a0de345f3a3be1971f6c729e8 P 625889083e23420fb414432cf88b0b51 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 09:26:35.129051 23997 raft_consensus.cc:482] T cb17be8a0de345f3a3be1971f6c729e8 P 625889083e23420fb414432cf88b0b51 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 09:26:35.129068 23997 raft_consensus.cc:3037] T cb17be8a0de345f3a3be1971f6c729e8 P 625889083e23420fb414432cf88b0b51 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:26:35.129796 23997 raft_consensus.cc:504] T cb17be8a0de345f3a3be1971f6c729e8 P 625889083e23420fb414432cf88b0b51 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625889083e23420fb414432cf88b0b51" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 37091 } } peers { permanent_uuid: "3a91f37200fc4719b1d4853f3fbaefe3" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 39089 } } peers { permanent_uuid: "318c1dc31882444aa5becb0236f255ea" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 38265 } }
I20260620 09:26:35.129953 23997 leader_election.cc:288] T cb17be8a0de345f3a3be1971f6c729e8 P 625889083e23420fb414432cf88b0b51 [CANDIDATE]: Term 1 election: Requested vote from peers 3a91f37200fc4719b1d4853f3fbaefe3 (127.16.43.131:39089), 318c1dc31882444aa5becb0236f255ea (127.16.43.130:38265)
I20260620 09:26:35.130087 23846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb17be8a0de345f3a3be1971f6c729e8" candidate_uuid: "625889083e23420fb414432cf88b0b51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a91f37200fc4719b1d4853f3fbaefe3"
I20260620 09:26:35.130127 23710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb17be8a0de345f3a3be1971f6c729e8" candidate_uuid: "625889083e23420fb414432cf88b0b51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "318c1dc31882444aa5becb0236f255ea"
I20260620 09:26:35.130180 23846 raft_consensus.cc:3037] T cb17be8a0de345f3a3be1971f6c729e8 P 3a91f37200fc4719b1d4853f3fbaefe3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:26:35.130198 23710 raft_consensus.cc:3037] T cb17be8a0de345f3a3be1971f6c729e8 P 318c1dc31882444aa5becb0236f255ea [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:26:35.130964 23846 raft_consensus.cc:2454] T cb17be8a0de345f3a3be1971f6c729e8 P 3a91f37200fc4719b1d4853f3fbaefe3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 625889083e23420fb414432cf88b0b51 in term 1.
I20260620 09:26:35.131114 23710 raft_consensus.cc:2454] T cb17be8a0de345f3a3be1971f6c729e8 P 318c1dc31882444aa5becb0236f255ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 625889083e23420fb414432cf88b0b51 in term 1.
I20260620 09:26:35.131175 23508 leader_election.cc:302] T cb17be8a0de345f3a3be1971f6c729e8 P 625889083e23420fb414432cf88b0b51 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a91f37200fc4719b1d4853f3fbaefe3, 625889083e23420fb414432cf88b0b51; no voters:
I20260620 09:26:35.131268 23997 raft_consensus.cc:2781] T cb17be8a0de345f3a3be1971f6c729e8 P 625889083e23420fb414432cf88b0b51 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:26:35.131466 23997 raft_consensus.cc:686] T cb17be8a0de345f3a3be1971f6c729e8 P 625889083e23420fb414432cf88b0b51 [term 1 LEADER]: Becoming Leader. State: Replica: 625889083e23420fb414432cf88b0b51, State: Running, Role: LEADER
I20260620 09:26:35.131567 23997 consensus_queue.cc:237] T cb17be8a0de345f3a3be1971f6c729e8 P 625889083e23420fb414432cf88b0b51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625889083e23420fb414432cf88b0b51" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 37091 } } peers { permanent_uuid: "3a91f37200fc4719b1d4853f3fbaefe3" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 39089 } } peers { permanent_uuid: "318c1dc31882444aa5becb0236f255ea" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 38265 } }
I20260620 09:26:35.132088 23972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:35.131992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:40995 (local address 127.16.43.129:39781)
0620 09:26:35.132032 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:35.132061 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:40995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260620 09:26:35.132462 23439 catalog_manager.cc:5495] T cb17be8a0de345f3a3be1971f6c729e8 P 625889083e23420fb414432cf88b0b51 reported cstate change: term changed from 0 to 1, leader changed from <none> to 625889083e23420fb414432cf88b0b51 (127.16.43.129). New cstate: current_term: 1 leader_uuid: "625889083e23420fb414432cf88b0b51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625889083e23420fb414432cf88b0b51" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 37091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a91f37200fc4719b1d4853f3fbaefe3" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 39089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "318c1dc31882444aa5becb0236f255ea" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 38265 } health_report { overall_health: UNKNOWN } } }
I20260620 09:26:35.172299 23987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:35.169758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34018 (local address 127.16.43.129:37091)
0620 09:26:35.169833 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:35.169835 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:35.169844 (+ 9us) server_negotiation.cc:407] Connection header received
0620 09:26:35.169885 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:35.169885 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:35.169911 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:35.169991 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:26:35.170134 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:35.170643 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:35.171424 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:35.171703 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:35.171783 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:35.171954 (+ 171us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:26:35.172066 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:26:35.172145 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:35.172189 (+ 44us) server_negotiation.cc:299] Negotiation successful
0620 09:26:35.172210 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 09:26:35.173492 23846 raft_consensus.cc:1261] T cb17be8a0de345f3a3be1971f6c729e8 P 3a91f37200fc4719b1d4853f3fbaefe3 [term 1 FOLLOWER]: Refusing update from remote peer 625889083e23420fb414432cf88b0b51: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:26:35.173579 23710 raft_consensus.cc:1261] T cb17be8a0de345f3a3be1971f6c729e8 P 318c1dc31882444aa5becb0236f255ea [term 1 FOLLOWER]: Refusing update from remote peer 625889083e23420fb414432cf88b0b51: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:26:35.173779 24001 consensus_queue.cc:1035] T cb17be8a0de345f3a3be1971f6c729e8 P 625889083e23420fb414432cf88b0b51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a91f37200fc4719b1d4853f3fbaefe3" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 39089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:26:35.173915 23997 consensus_queue.cc:1035] T cb17be8a0de345f3a3be1971f6c729e8 P 625889083e23420fb414432cf88b0b51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "318c1dc31882444aa5becb0236f255ea" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 38265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:26:35.174661 23971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:35.174547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:40995 (local address 127.16.43.131:47215)
0620 09:26:35.174588 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:35.174623 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:40995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260620 09:26:35.175004 23973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:35.174870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:40995 (local address 127.16.43.130:36563)
0620 09:26:35.174939 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:35.174970 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:40995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260620 09:26:35.176057 23972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:35.175857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:40995 (local address 127.16.43.129:45577)
0620 09:26:35.176001 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:35.176028 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:40995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130}
I20260620 09:26:35.175902 24006 mvcc.cc:205] Tried to move back new op lower bound from 7298857349828534272 to 7298857349659484160. Current Snapshot: MvccSnapshot[applied={T|T < 7298857349828534272}]
I20260620 09:26:35.176265 23971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:35.176172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:40995 (local address 127.16.43.131:42865)
0620 09:26:35.176213 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:35.176238 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:40995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260620 09:26:35.177449 23972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:35.177346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:40995 (local address 127.16.43.129:55177)
0620 09:26:35.177379 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:35.177419 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:40995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20260620 09:26:35.177889 23973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:35.177800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:40995 (local address 127.16.43.130:58517)
0620 09:26:35.177835 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:35.177860 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:40995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260620 09:26:35.178535 23439 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36222:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:26:35.178644 23439 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36222:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:26:35.180004 23439 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fc318f0d33f1422abfc6e301edf09951: Sending DeleteTablet for 3 replicas of tablet cb17be8a0de345f3a3be1971f6c729e8
I20260620 09:26:35.180392 23553 tablet_service.cc:1514] Processing DeleteTablet for tablet cb17be8a0de345f3a3be1971f6c729e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:26:35 UTC) from {username='oryx', principal='oryx/127.16.43.188@KRBTEST.COM'} at 127.0.0.1:34004
I20260620 09:26:35.180420 23690 tablet_service.cc:1514] Processing DeleteTablet for tablet cb17be8a0de345f3a3be1971f6c729e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:26:35 UTC) from {username='oryx', principal='oryx/127.16.43.188@KRBTEST.COM'} at 127.0.0.1:33114
I20260620 09:26:35.180617 24016 tablet_replica.cc:330] T cb17be8a0de345f3a3be1971f6c729e8 P 625889083e23420fb414432cf88b0b51: stopping tablet replica
I20260620 09:26:35.180670 24016 raft_consensus.cc:2229] T cb17be8a0de345f3a3be1971f6c729e8 P 625889083e23420fb414432cf88b0b51 [term 1 LEADER]: Raft consensus shutting down.
I20260620 09:26:35.180764 24016 raft_consensus.cc:2258] T cb17be8a0de345f3a3be1971f6c729e8 P 625889083e23420fb414432cf88b0b51 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:26:35.180757 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 23401
I20260620 09:26:35.181397 24015 tablet_replica.cc:330] T cb17be8a0de345f3a3be1971f6c729e8 P 318c1dc31882444aa5becb0236f255ea: stopping tablet replica
I20260620 09:26:35.181452 24015 raft_consensus.cc:2229] T cb17be8a0de345f3a3be1971f6c729e8 P 318c1dc31882444aa5becb0236f255ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:26:35.181499 24015 raft_consensus.cc:2258] T cb17be8a0de345f3a3be1971f6c729e8 P 318c1dc31882444aa5becb0236f255ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:26:35.181849 24015 ts_tablet_manager.cc:1905] T cb17be8a0de345f3a3be1971f6c729e8 P 318c1dc31882444aa5becb0236f255ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:26:35.182775 23826 tablet_service.cc:1514] Processing DeleteTablet for tablet cb17be8a0de345f3a3be1971f6c729e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:26:35 UTC) from {username='oryx', principal='oryx/127.16.43.188@KRBTEST.COM'} at 127.0.0.1:59890
I20260620 09:26:35.182991 24017 tablet_replica.cc:330] T cb17be8a0de345f3a3be1971f6c729e8 P 3a91f37200fc4719b1d4853f3fbaefe3: stopping tablet replica
I20260620 09:26:35.183077 24017 raft_consensus.cc:2229] T cb17be8a0de345f3a3be1971f6c729e8 P 3a91f37200fc4719b1d4853f3fbaefe3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:26:35.183132 24017 raft_consensus.cc:2258] T cb17be8a0de345f3a3be1971f6c729e8 P 3a91f37200fc4719b1d4853f3fbaefe3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:26:35.183418 24017 ts_tablet_manager.cc:1905] T cb17be8a0de345f3a3be1971f6c729e8 P 3a91f37200fc4719b1d4853f3fbaefe3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:26:35.183985 24015 ts_tablet_manager.cc:1918] T cb17be8a0de345f3a3be1971f6c729e8 P 318c1dc31882444aa5becb0236f255ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:26:35.184055 24015 log.cc:1196] T cb17be8a0de345f3a3be1971f6c729e8 P 318c1dc31882444aa5becb0236f255ea: Deleting WAL directory at /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/ts-1/wal/wals/cb17be8a0de345f3a3be1971f6c729e8
I20260620 09:26:35.184345 24015 ts_tablet_manager.cc:1939] T cb17be8a0de345f3a3be1971f6c729e8 P 318c1dc31882444aa5becb0236f255ea: Deleting consensus metadata
I20260620 09:26:35.184722 23427 catalog_manager.cc:4841] TS 318c1dc31882444aa5becb0236f255ea (127.16.43.130:38265): tablet cb17be8a0de345f3a3be1971f6c729e8 (table test-table [id=8fda61ba944c407b98a6a9e183971295]) successfully deleted
I20260620 09:26:35.185123 24017 ts_tablet_manager.cc:1918] T cb17be8a0de345f3a3be1971f6c729e8 P 3a91f37200fc4719b1d4853f3fbaefe3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:26:35.185222 24017 log.cc:1196] T cb17be8a0de345f3a3be1971f6c729e8 P 3a91f37200fc4719b1d4853f3fbaefe3: Deleting WAL directory at /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781947562949448-16558-0/minicluster-data/ts-2/wal/wals/cb17be8a0de345f3a3be1971f6c729e8
I20260620 09:26:35.185508 24017 ts_tablet_manager.cc:1939] T cb17be8a0de345f3a3be1971f6c729e8 P 3a91f37200fc4719b1d4853f3fbaefe3: Deleting consensus metadata
I20260620 09:26:35.185828 23426 catalog_manager.cc:4841] TS 3a91f37200fc4719b1d4853f3fbaefe3 (127.16.43.131:39089): tablet cb17be8a0de345f3a3be1971f6c729e8 (table test-table [id=8fda61ba944c407b98a6a9e183971295]) successfully deleted
I20260620 09:26:35.187949 23988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:35.187774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.129:37091 (local address 127.0.0.1:34026)
0620 09:26:35.187879 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:35.187913 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.129:37091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260620 09:26:35.187935 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 23627
W20260620 09:26:35.188643 23427 catalog_manager.cc:4584] TS 625889083e23420fb414432cf88b0b51 (127.16.43.129:37091): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cb17be8a0de345f3a3be1971f6c729e8: Network error: Client connection negotiation failed: client connection to 127.16.43.129:37091: connect: Connection refused (error 111)
I20260620 09:26:35.193835 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 23763
I20260620 09:26:35.198861 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 23121
I20260620 09:26:35.203452 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 23308
2026-06-20T09:26:35Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (6836 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781947562949448-16558-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:26:35 dist-test-slave-706q krb5kdc[24020](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:26:35 dist-test-slave-706q krb5kdc[24020](info): set up 2 sockets
Jun 20 09:26:35 dist-test-slave-706q krb5kdc[24020](info): commencing operation
krb5kdc: starting...
W20260620 09:26:36.249091 16558 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781947562949448-16558-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781947562949448-16558-0/krb5kdc/test-user.keytab.
Jun 20 09:26:36 dist-test-slave-706q krb5kdc[24020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947596, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:26:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:26:36Z Disabled control of system clock
WARNING: no policy specified for oryx/127.16.43.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.43.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.43.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781947562949448-16558-0/krb5kdc/oryx_127.16.43.190.keytab.
Entry for principal oryx/127.16.43.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781947562949448-16558-0/krb5kdc/oryx_127.16.43.190.keytab.
I20260620 09:26:36.374034 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781947562949448-16558-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781947562949448-16558-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781947562949448-16558-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.43.190:36561
--webserver_interface=127.16.43.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781947562949448-16558-0/krb5kdc/oryx_127.16.43.190.keytab
--principal=oryx/127.16.43.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.43.148:33371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.43.190:36561
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781947562949448-16558-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781947562949448-16558-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781947562949448-16558-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:26:36.457756 24033 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:26:36.458024 24033 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:26:36.458063 24033 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:26:36.459252 24033 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:26:36.459326 24033 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 09:26:36.459357 24033 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:26:36.459391 24033 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:26:36.459422 24033 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:26:36.459444 24033 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:26:36.461103 24033 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:33371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781947562949448-16558-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781947562949448-16558-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.43.190:36561
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781947562949448-16558-0/krb5kdc/oryx_127.16.43.190.keytab
--principal=oryx/127.16.43.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.43.190:36561
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.43.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24033
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781947562949448-16558-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:26:36.461403 24033 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:26:36.461650 24033 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:26:36.466955 24041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:26:36.467106 24033 server_base.cc:1034] running on GCE node
W20260620 09:26:36.466928 24038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:26:36.466945 24039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:26:36.467502 24033 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:26:36.467823 24033 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:26:36.468981 24033 hybrid_clock.cc:648] HybridClock initialized: now 1781947596468902 us; error 77 us; skew 500 ppm
Jun 20 09:26:36 dist-test-slave-706q krb5kdc[24020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947596, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.43.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:26:36.471966 24033 init.cc:377] Logged in from keytab as oryx/127.16.43.190@KRBTEST.COM (short username oryx)
I20260620 09:26:36.472473 24033 webserver.cc:460] Webserver started at http://127.16.43.190:32871/ using document root <none> and password file <none>
I20260620 09:26:36.472720 24033 fs_manager.cc:359] Metadata directory not provided
I20260620 09:26:36.472795 24033 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:26:36.472961 24033 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:26:36.474066 24033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781947562949448-16558-0/minicluster-data/master-0/data/instance:
uuid: "d0f24a0b0e524fd884b6522ce4569de5"
format_stamp: "Formatted at 2026-06-20 09:26:36 on dist-test-slave-706q"
server_key: "c187aa1c08138a9c3e54c3b3240388b8"
server_key_iv: "c27a4bc0f0870d53bdd65619e77c75a3"
server_key_version: "encryptionkey@0"
I20260620 09:26:36.474457 24033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781947562949448-16558-0/minicluster-data/master-0/wal/instance:
uuid: "d0f24a0b0e524fd884b6522ce4569de5"
format_stamp: "Formatted at 2026-06-20 09:26:36 on dist-test-slave-706q"
server_key: "c187aa1c08138a9c3e54c3b3240388b8"
server_key_iv: "c27a4bc0f0870d53bdd65619e77c75a3"
server_key_version: "encryptionkey@0"
I20260620 09:26:36.475930 24033 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 09:26:36.476771 24048 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:36.476986 24033 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 09:26:36.477088 24033 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781947562949448-16558-0/minicluster-data/master-0/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781947562949448-16558-0/minicluster-data/master-0/wal
uuid: "d0f24a0b0e524fd884b6522ce4569de5"
format_stamp: "Formatted at 2026-06-20 09:26:36 on dist-test-slave-706q"
server_key: "c187aa1c08138a9c3e54c3b3240388b8"
server_key_iv: "c27a4bc0f0870d53bdd65619e77c75a3"
server_key_version: "encryptionkey@0"
I20260620 09:26:36.477173 24033 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781947562949448-16558-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781947562949448-16558-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781947562949448-16558-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:26:36.492882 24033 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:26:36.495294 24033 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:26:36.495481 24033 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:26:36.500267 24033 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.190:36561
I20260620 09:26:36.500339 24110 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.190:36561 every 8 connection(s)
I20260620 09:26:36.500712 24033 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
I20260620 09:26:36.501405 24111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:26:36.503942 24111 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0f24a0b0e524fd884b6522ce4569de5: Bootstrap starting.
I20260620 09:26:36.504595 24111 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d0f24a0b0e524fd884b6522ce4569de5: Neither blocks nor log segments found. Creating new log.
I20260620 09:26:36.504910 24111 log.cc:824] T 00000000000000000000000000000000 P d0f24a0b0e524fd884b6522ce4569de5: Log is configured to *not* fsync() on all Append() calls
I20260620 09:26:36.505666 24111 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0f24a0b0e524fd884b6522ce4569de5: No bootstrap required, opened a new log
I20260620 09:26:36.507834 24111 raft_consensus.cc:348] T 00000000000000000000000000000000 P d0f24a0b0e524fd884b6522ce4569de5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0f24a0b0e524fd884b6522ce4569de5" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 36561 } }
I20260620 09:26:36.507978 24111 raft_consensus.cc:374] T 00000000000000000000000000000000 P d0f24a0b0e524fd884b6522ce4569de5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:26:36.508001 24111 raft_consensus.cc:729] T 00000000000000000000000000000000 P d0f24a0b0e524fd884b6522ce4569de5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0f24a0b0e524fd884b6522ce4569de5, State: Initialized, Role: FOLLOWER
I20260620 09:26:36.508188 24111 consensus_queue.cc:260] T 00000000000000000000000000000000 P d0f24a0b0e524fd884b6522ce4569de5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0f24a0b0e524fd884b6522ce4569de5" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 36561 } }
I20260620 09:26:36.508250 24111 raft_consensus.cc:388] T 00000000000000000000000000000000 P d0f24a0b0e524fd884b6522ce4569de5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:26:36.508294 24111 raft_consensus.cc:482] T 00000000000000000000000000000000 P d0f24a0b0e524fd884b6522ce4569de5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:26:36.508342 24111 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d0f24a0b0e524fd884b6522ce4569de5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:26:36.508917 24111 raft_consensus.cc:504] T 00000000000000000000000000000000 P d0f24a0b0e524fd884b6522ce4569de5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0f24a0b0e524fd884b6522ce4569de5" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 36561 } }
I20260620 09:26:36.509032 24111 leader_election.cc:302] T 00000000000000000000000000000000 P d0f24a0b0e524fd884b6522ce4569de5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d0f24a0b0e524fd884b6522ce4569de5; no voters:
I20260620 09:26:36.509205 24111 leader_election.cc:288] T 00000000000000000000000000000000 P d0f24a0b0e524fd884b6522ce4569de5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:26:36.509264 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 24033
I20260620 09:26:36.509272 24114 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d0f24a0b0e524fd884b6522ce4569de5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:26:36.509372 16558 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781947562949448-16558-0/minicluster-data/master-0/wal/instance
I20260620 09:26:36.509446 24114 raft_consensus.cc:686] T 00000000000000000000000000000000 P d0f24a0b0e524fd884b6522ce4569de5 [term 1 LEADER]: Becoming Leader. State: Replica: d0f24a0b0e524fd884b6522ce4569de5, State: Running, Role: LEADER
I20260620 09:26:36.509501 24111 sys_catalog.cc:564] T 00000000000000000000000000000000 P d0f24a0b0e524fd884b6522ce4569de5 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:26:36.509543 24114 consensus_queue.cc:237] T 00000000000000000000000000000000 P d0f24a0b0e524fd884b6522ce4569de5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0f24a0b0e524fd884b6522ce4569de5" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 36561 } }
I20260620 09:26:36.509603 16558 external_mini_cluster.cc:1428] Setting key ebad80362239a0b6147ee9990e29a292
I20260620 09:26:36.509861 24115 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0f24a0b0e524fd884b6522ce4569de5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d0f24a0b0e524fd884b6522ce4569de5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0f24a0b0e524fd884b6522ce4569de5" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 36561 } } }
I20260620 09:26:36.509960 24115 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0f24a0b0e524fd884b6522ce4569de5 [sys.catalog]: This master's current role is: LEADER
I20260620 09:26:36.510129 24116 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0f24a0b0e524fd884b6522ce4569de5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0f24a0b0e524fd884b6522ce4569de5. Latest consensus state: current_term: 1 leader_uuid: "d0f24a0b0e524fd884b6522ce4569de5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0f24a0b0e524fd884b6522ce4569de5" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 36561 } } }
I20260620 09:26:36.510200 24116 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0f24a0b0e524fd884b6522ce4569de5 [sys.catalog]: This master's current role is: LEADER
I20260620 09:26:36.510334 24120 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:26:36.510923 24120 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:26:36.512292 24133 mvcc.cc:205] Tried to move back new op lower bound from 7298857355312373760 to 7298857355303292928. Current Snapshot: MvccSnapshot[applied={T|T < 7298857355312373760}]
I20260620 09:26:36.512737 24120 catalog_manager.cc:1347] Generated new cluster ID: 783ef1dcc4e2498a8060fd964d37e459
I20260620 09:26:36.512784 24120 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 09:26:36 dist-test-slave-706q krb5kdc[24020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947596, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.43.190@KRBTEST.COM
I20260620 09:26:36.519984 24120 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:26:36.520681 24120 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:26:36.521553 24125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:36.510258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48028 (local address 127.16.43.190:36561)
0620 09:26:36.510578 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:36.510588 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:36.510624 (+ 36us) server_negotiation.cc:407] Connection header received
0620 09:26:36.510748 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:36.510756 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:36.510813 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:36.510960 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:36.511752 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:36.513010 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:36.514055 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:36.514675 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:36.514770 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:36.517690 (+ 2920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:36.517698 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:36.517698 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:26:36.517715 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:36.519829 (+ 2114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:36.520251 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:36.520254 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:36.520254 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:36.520381 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:36.520612 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:36.520613 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:36.520614 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:36.520898 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:36.521049 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:36.521174 (+ 125us) server_negotiation.cc:299] Negotiation successful
0620 09:26:36.521319 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":60,"threads_started":1}
I20260620 09:26:36.524238 24120 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d0f24a0b0e524fd884b6522ce4569de5: Generated new TSK 0
I20260620 09:26:36.524413 24120 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.16.43.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.43.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.43.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781947562949448-16558-0/krb5kdc/oryx_127.16.43.129.keytab.
Entry for principal oryx/127.16.43.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781947562949448-16558-0/krb5kdc/oryx_127.16.43.129.keytab.
I20260620 09:26:36.562861 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781947562949448-16558-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781947562949448-16558-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781947562949448-16558-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781947562949448-16558-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.43.129:0
--local_ip_for_outbound_sockets=127.16.43.129
--webserver_interface=127.16.43.129
--webserver_port=0
--tserver_master_addrs=127.16.43.190:36561
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781947562949448-16558-0/krb5kdc/oryx_127.16.43.129.keytab
--principal=oryx/127.16.43.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.43.148:33371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781947562949448-16558-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781947562949448-16558-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781947562949448-16558-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:26:36.646450 24137 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:26:36.646708 24137 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:26:36.646751 24137 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:26:36.648007 24137 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:26:36.648084 24137 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:26:36.648124 24137 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 09:26:36.648162 24137 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.43.129
I20260620 09:26:36.649762 24137 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:33371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781947562949448-16558-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781947562949448-16558-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781947562949448-16558-0/krb5kdc/oryx_127.16.43.129.keytab
--principal=oryx/127.16.43.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.43.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781947562949448-16558-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.43.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.43.190:36561
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24137
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.43.129
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781947562949448-16558-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:26:36.650048 24137 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:26:36.650296 24137 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:26:36.655613 24143 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:26:36.655613 24145 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:26:36.655642 24142 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:26:36.655766 24137 server_base.cc:1034] running on GCE node
I20260620 09:26:36.656052 24137 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:26:36.656299 24137 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:26:36.657472 24137 hybrid_clock.cc:648] HybridClock initialized: now 1781947596657454 us; error 37 us; skew 500 ppm
Jun 20 09:26:36 dist-test-slave-706q krb5kdc[24020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947596, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.43.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:26:36.659986 24137 init.cc:377] Logged in from keytab as oryx/127.16.43.129@KRBTEST.COM (short username oryx)
I20260620 09:26:36.660488 24137 webserver.cc:460] Webserver started at http://127.16.43.129:41243/ using document root <none> and password file <none>
I20260620 09:26:36.660755 24137 fs_manager.cc:359] Metadata directory not provided
I20260620 09:26:36.660842 24137 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:26:36.661002 24137 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:26:36.662063 24137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781947562949448-16558-0/minicluster-data/ts-0/data/instance:
uuid: "6b68c749403d4f96a817a83a1a679a8d"
format_stamp: "Formatted at 2026-06-20 09:26:36 on dist-test-slave-706q"
server_key: "a2caa51d8425780a076526b4ddb307cf"
server_key_iv: "a89f339bbe4c0111033ab99564ed4a69"
server_key_version: "encryptionkey@0"
I20260620 09:26:36.662441 24137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781947562949448-16558-0/minicluster-data/ts-0/wal/instance:
uuid: "6b68c749403d4f96a817a83a1a679a8d"
format_stamp: "Formatted at 2026-06-20 09:26:36 on dist-test-slave-706q"
server_key: "a2caa51d8425780a076526b4ddb307cf"
server_key_iv: "a89f339bbe4c0111033ab99564ed4a69"
server_key_version: "encryptionkey@0"
I20260620 09:26:36.663851 24137 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 09:26:36.664556 24152 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:36.664724 24137 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 09:26:36.664829 24137 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781947562949448-16558-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781947562949448-16558-0/minicluster-data/ts-0/wal
uuid: "6b68c749403d4f96a817a83a1a679a8d"
format_stamp: "Formatted at 2026-06-20 09:26:36 on dist-test-slave-706q"
server_key: "a2caa51d8425780a076526b4ddb307cf"
server_key_iv: "a89f339bbe4c0111033ab99564ed4a69"
server_key_version: "encryptionkey@0"
I20260620 09:26:36.664904 24137 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781947562949448-16558-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781947562949448-16558-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781947562949448-16558-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:26:36.681198 24137 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:26:36.683369 24137 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:26:36.683643 24137 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:26:36.684306 24137 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:26:36.684367 24137 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:36.684399 24137 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:26:36.684420 24137 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 09:26:36 dist-test-slave-706q krb5kdc[24020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947596, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.43.129@KRBTEST.COM for oryx/127.16.43.190@KRBTEST.COM
I20260620 09:26:36.691781 24137 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.129:39361
I20260620 09:26:36.691855 24267 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.129:39361 every 8 connection(s)
I20260620 09:26:36.692212 24137 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781947562949448-16558-0/minicluster-data/ts-0/data/info.pb
I20260620 09:26:36.694635 24125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:36.684790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.129:50815 (local address 127.16.43.190:36561)
0620 09:26:36.684859 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:36.684862 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:36.685083 (+ 221us) server_negotiation.cc:407] Connection header received
0620 09:26:36.685669 (+ 586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:36.685670 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:36.685683 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:36.685745 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:36.686438 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:36.687307 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:36.688565 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:36.688805 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:36.688889 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:36.691870 (+ 2981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:36.691872 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:36.691873 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:26:36.691882 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:36.693410 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:36.693829 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:36.693830 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:36.693831 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:36.693906 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:36.694133 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:36.694133 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:36.694134 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:36.694248 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:36.694306 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:36.694502 (+ 196us) server_negotiation.cc:299] Negotiation successful
0620 09:26:36.694569 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 09:26:36.694875 24161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:36.684749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:36561 (local address 127.16.43.129:50815)
0620 09:26:36.684955 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:36.684995 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:36.685546 (+ 551us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:36.685817 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:36.685822 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:36.686143 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:36.686333 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:36.686337 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:36.687490 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:36.687491 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:36.688453 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:36.688456 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:36.688917 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:36.688918 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:36.689041 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:36.689634 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:26:36.689642 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:26:36.691734 (+ 2092us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:26:36.693483 (+ 1749us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:36.693489 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:36.693491 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:36.693714 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:36.693975 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:36.693976 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:36.693976 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:36.694073 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:36.694297 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:26:36.694299 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:26:36.694410 (+ 111us) client_negotiation.cc:769] Sending connection context
0620 09:26:36.694491 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 09:26:36.694612 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":81,"threads_started":1}
I20260620 09:26:36.695367 24268 heartbeater.cc:347] Connected to a master server at 127.16.43.190:36561
I20260620 09:26:36.695447 24268 heartbeater.cc:464] Registering TS with master...
I20260620 09:26:36.695674 24268 heartbeater.cc:511] Master 127.16.43.190:36561 requested a full tablet report, sending...
I20260620 09:26:36.696241 24064 ts_manager.cc:194] Registered new tserver with Master: 6b68c749403d4f96a817a83a1a679a8d (127.16.43.129:39361)
I20260620 09:26:36.697069 24064 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.43.129@KRBTEST.COM'} at 127.16.43.129:50815
I20260620 09:26:36.698218 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 24137
I20260620 09:26:36.698302 16558 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781947562949448-16558-0/minicluster-data/ts-0/wal/instance
I20260620 09:26:36.698524 16558 external_mini_cluster.cc:1428] Setting key 88e08f37ae0f52202d4f0c9ef7992de5
I20260620 09:26:36.702327 24125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:36.696531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.129:57749 (local address 127.16.43.190:36561)
0620 09:26:36.696570 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:36.696572 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:36.696638 (+ 66us) server_negotiation.cc:407] Connection header received
0620 09:26:36.696699 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:36.696700 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:36.696709 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:36.696760 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:36.697043 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:36.697695 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:36.698443 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:36.698695 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:36.698767 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:36.699914 (+ 1147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:36.699915 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:36.699916 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:26:36.699923 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:36.701305 (+ 1382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:36.701689 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:36.701690 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:36.701690 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:36.701736 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:36.701922 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:36.701922 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:36.701922 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:36.702010 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:36.702060 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:36.702216 (+ 156us) server_negotiation.cc:299] Negotiation successful
0620 09:26:36.702267 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 09:26:36.702596 24274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:36.696460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:36561 (local address 127.16.43.129:57749)
0620 09:26:36.696578 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:36.696588 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:36.696647 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:36.696775 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:36.696775 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:36.696922 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:36.696977 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:36.696978 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:36.697791 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:36.697792 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:36.698379 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:36.698381 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:36.698901 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:36.698902 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:36.698978 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:36.699467 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:26:36.699472 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:26:36.699820 (+ 348us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:26:36.701386 (+ 1566us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:36.701387 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:36.701388 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:36.701616 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:36.701793 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:36.701794 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:36.701794 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:36.701865 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:36.702074 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:26:36.702075 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:26:36.702124 (+ 49us) client_negotiation.cc:769] Sending connection context
0620 09:26:36.702202 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 09:26:36.702315 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":67,"threads_started":1}
WARNING: no policy specified for oryx/127.16.43.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.43.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.43.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781947562949448-16558-0/krb5kdc/oryx_127.16.43.130.keytab.
Entry for principal oryx/127.16.43.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781947562949448-16558-0/krb5kdc/oryx_127.16.43.130.keytab.
I20260620 09:26:36.726960 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781947562949448-16558-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781947562949448-16558-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781947562949448-16558-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781947562949448-16558-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.43.130:0
--local_ip_for_outbound_sockets=127.16.43.130
--webserver_interface=127.16.43.130
--webserver_port=0
--tserver_master_addrs=127.16.43.190:36561
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781947562949448-16558-0/krb5kdc/oryx_127.16.43.130.keytab
--principal=oryx/127.16.43.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.43.148:33371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781947562949448-16558-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781947562949448-16558-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781947562949448-16558-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:26:36.809777 24277 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:26:36.810020 24277 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:26:36.810046 24277 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:26:36.811264 24277 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:26:36.811335 24277 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:26:36.811372 24277 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 09:26:36.811403 24277 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.43.130
I20260620 09:26:36.812976 24277 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:33371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781947562949448-16558-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781947562949448-16558-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781947562949448-16558-0/krb5kdc/oryx_127.16.43.130.keytab
--principal=oryx/127.16.43.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.43.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781947562949448-16558-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.43.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.43.190:36561
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24277
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.43.130
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781947562949448-16558-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:26:36.813256 24277 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:26:36.813500 24277 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:26:36.818821 24283 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:26:36.818861 24277 server_base.cc:1034] running on GCE node
W20260620 09:26:36.818830 24282 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:26:36.819037 24285 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:26:36.819258 24277 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:26:36.819562 24277 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:26:36.820813 24277 hybrid_clock.cc:648] HybridClock initialized: now 1781947596820793 us; error 39 us; skew 500 ppm
Jun 20 09:26:36 dist-test-slave-706q krb5kdc[24020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947596, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.43.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:26:36.823504 24277 init.cc:377] Logged in from keytab as oryx/127.16.43.130@KRBTEST.COM (short username oryx)
I20260620 09:26:36.824092 24277 webserver.cc:460] Webserver started at http://127.16.43.130:42345/ using document root <none> and password file <none>
I20260620 09:26:36.824342 24277 fs_manager.cc:359] Metadata directory not provided
I20260620 09:26:36.824411 24277 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:26:36.824561 24277 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:26:36.825785 24277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781947562949448-16558-0/minicluster-data/ts-1/data/instance:
uuid: "6b9304cb76a2417b956f9e63e78b7734"
format_stamp: "Formatted at 2026-06-20 09:26:36 on dist-test-slave-706q"
server_key: "9fa54d5e1cb56d496195d5ad204a5f15"
server_key_iv: "92a54f24cac73ed2c6af09e936f7106f"
server_key_version: "encryptionkey@0"
I20260620 09:26:36.826155 24277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781947562949448-16558-0/minicluster-data/ts-1/wal/instance:
uuid: "6b9304cb76a2417b956f9e63e78b7734"
format_stamp: "Formatted at 2026-06-20 09:26:36 on dist-test-slave-706q"
server_key: "9fa54d5e1cb56d496195d5ad204a5f15"
server_key_iv: "92a54f24cac73ed2c6af09e936f7106f"
server_key_version: "encryptionkey@0"
I20260620 09:26:36.827574 24277 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 09:26:36.828389 24292 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:36.828575 24277 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:26:36.828675 24277 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781947562949448-16558-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781947562949448-16558-0/minicluster-data/ts-1/wal
uuid: "6b9304cb76a2417b956f9e63e78b7734"
format_stamp: "Formatted at 2026-06-20 09:26:36 on dist-test-slave-706q"
server_key: "9fa54d5e1cb56d496195d5ad204a5f15"
server_key_iv: "92a54f24cac73ed2c6af09e936f7106f"
server_key_version: "encryptionkey@0"
I20260620 09:26:36.828756 24277 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781947562949448-16558-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781947562949448-16558-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781947562949448-16558-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:26:36.841058 24277 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:26:36.843577 24277 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:26:36.843726 24277 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:26:36.844277 24277 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:26:36.844343 24277 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:36.844404 24277 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:26:36.844429 24277 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:36.850965 24277 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.130:35987
I20260620 09:26:36.851078 24407 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.130:35987 every 8 connection(s)
I20260620 09:26:36.851447 24277 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781947562949448-16558-0/minicluster-data/ts-1/data/info.pb
Jun 20 09:26:36 dist-test-slave-706q krb5kdc[24020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947596, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.43.130@KRBTEST.COM for oryx/127.16.43.190@KRBTEST.COM
I20260620 09:26:36.852608 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 24277
I20260620 09:26:36.852715 16558 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781947562949448-16558-0/minicluster-data/ts-1/wal/instance
I20260620 09:26:36.852941 16558 external_mini_cluster.cc:1428] Setting key b58f6774369f47634bbfff870a60753f
I20260620 09:26:36.856096 24125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:36.844892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.130:39029 (local address 127.16.43.190:36561)
0620 09:26:36.844974 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:36.844977 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:36.845255 (+ 278us) server_negotiation.cc:407] Connection header received
0620 09:26:36.845920 (+ 665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:36.845922 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:36.845939 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:36.845974 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:36.846998 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:36.847841 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:36.849622 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:36.849999 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:36.850081 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:36.852858 (+ 2777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:36.852860 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:36.852861 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:26:36.852909 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:36.854489 (+ 1580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:36.855020 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:36.855022 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:36.855022 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:36.855090 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:36.855327 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:36.855327 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:36.855328 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:36.855493 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:36.855598 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:36.855861 (+ 263us) server_negotiation.cc:299] Negotiation successful
0620 09:26:36.855956 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 09:26:36.856115 24301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:36.844861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:36561 (local address 127.16.43.130:39029)
0620 09:26:36.845144 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:36.845174 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:36.845780 (+ 606us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:36.846184 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:36.846189 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:36.846573 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:36.846870 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:36.846877 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:36.847992 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:36.847994 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:36.849498 (+ 1504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:36.849502 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:36.850102 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:36.850103 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:36.850282 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:36.851030 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:26:36.851041 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:26:36.852738 (+ 1697us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:26:36.854580 (+ 1842us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:36.854588 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:36.854591 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:36.854933 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:36.855161 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:36.855161 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:36.855161 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:36.855262 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:36.855588 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:26:36.855594 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:26:36.855723 (+ 129us) client_negotiation.cc:769] Sending connection context
0620 09:26:36.855811 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 09:26:36.855948 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":116,"threads_started":1}
I20260620 09:26:36.856715 24408 heartbeater.cc:347] Connected to a master server at 127.16.43.190:36561
I20260620 09:26:36.856812 24408 heartbeater.cc:464] Registering TS with master...
I20260620 09:26:36.857007 24408 heartbeater.cc:511] Master 127.16.43.190:36561 requested a full tablet report, sending...
I20260620 09:26:36.857395 24064 ts_manager.cc:194] Registered new tserver with Master: 6b9304cb76a2417b956f9e63e78b7734 (127.16.43.130:35987)
I20260620 09:26:36.857909 24064 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.43.130@KRBTEST.COM'} at 127.16.43.130:39029
WARNING: no policy specified for oryx/127.16.43.131@KRBTEST.COM; defaulting to no policy
I20260620 09:26:36.863766 24125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:36.857806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.130:57447 (local address 127.16.43.190:36561)
0620 09:26:36.857841 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:36.857843 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:36.858159 (+ 316us) server_negotiation.cc:407] Connection header received
0620 09:26:36.858231 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:36.858232 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:36.858242 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:36.858288 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:36.858720 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:36.859226 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:36.859986 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:36.860262 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:36.860360 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:36.861338 (+ 978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:36.861339 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:36.861340 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:26:36.861347 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:36.862572 (+ 1225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:36.862921 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:36.862922 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:36.862923 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:36.862979 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:36.863182 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:36.863183 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:36.863184 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:36.863333 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:36.863416 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:36.863589 (+ 173us) server_negotiation.cc:299] Negotiation successful
0620 09:26:36.863669 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 09:26:36.863782 24415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:36.857748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:36561 (local address 127.16.43.130:57447)
0620 09:26:36.858107 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:36.858119 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:36.858174 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:36.858332 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:36.858333 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:36.858543 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:36.858616 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:36.858618 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:36.859318 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:36.859319 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:36.859903 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:36.859905 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:36.860368 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:36.860369 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:36.860442 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:36.860943 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:26:36.860948 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:26:36.861228 (+ 280us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:26:36.862659 (+ 1431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:36.862660 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:36.862661 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:36.862833 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:36.863051 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:36.863051 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:36.863052 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:36.863117 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:36.863408 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:26:36.863409 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:26:36.863457 (+ 48us) client_negotiation.cc:769] Sending connection context
0620 09:26:36.863570 (+ 113us) client_negotiation.cc:240] Negotiation successful
0620 09:26:36.863671 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":81,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.43.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.43.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781947562949448-16558-0/krb5kdc/oryx_127.16.43.131.keytab.
Entry for principal oryx/127.16.43.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781947562949448-16558-0/krb5kdc/oryx_127.16.43.131.keytab.
I20260620 09:26:36.878885 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781947562949448-16558-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781947562949448-16558-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781947562949448-16558-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781947562949448-16558-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.43.131:0
--local_ip_for_outbound_sockets=127.16.43.131
--webserver_interface=127.16.43.131
--webserver_port=0
--tserver_master_addrs=127.16.43.190:36561
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781947562949448-16558-0/krb5kdc/oryx_127.16.43.131.keytab
--principal=oryx/127.16.43.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.43.148:33371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781947562949448-16558-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781947562949448-16558-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781947562949448-16558-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:26:36.962726 24417 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:26:36.963011 24417 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:26:36.963052 24417 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:26:36.964336 24417 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:26:36.964417 24417 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:26:36.964452 24417 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 09:26:36.964494 24417 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.43.131
I20260620 09:26:36.966063 24417 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:33371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781947562949448-16558-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781947562949448-16558-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781947562949448-16558-0/krb5kdc/oryx_127.16.43.131.keytab
--principal=oryx/127.16.43.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.43.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781947562949448-16558-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.43.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.43.190:36561
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24417
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.43.131
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781947562949448-16558-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:26:36.966349 24417 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:26:36.966601 24417 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:26:36.971889 24423 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:26:36.971921 24422 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:26:36.971920 24425 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:26:36.972270 24417 server_base.cc:1034] running on GCE node
I20260620 09:26:36.972500 24417 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:26:36.972723 24417 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:26:36.973898 24417 hybrid_clock.cc:648] HybridClock initialized: now 1781947596973883 us; error 29 us; skew 500 ppm
Jun 20 09:26:36 dist-test-slave-706q krb5kdc[24020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947596, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.43.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:26:36.976613 24417 init.cc:377] Logged in from keytab as oryx/127.16.43.131@KRBTEST.COM (short username oryx)
I20260620 09:26:36.977027 24417 webserver.cc:460] Webserver started at http://127.16.43.131:43177/ using document root <none> and password file <none>
I20260620 09:26:36.977227 24417 fs_manager.cc:359] Metadata directory not provided
I20260620 09:26:36.977268 24417 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:26:36.977360 24417 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:26:36.978394 24417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781947562949448-16558-0/minicluster-data/ts-2/data/instance:
uuid: "41d4efe5a7fd4b2a94dd1489ff6f3482"
format_stamp: "Formatted at 2026-06-20 09:26:36 on dist-test-slave-706q"
server_key: "b4977a760701c0bfe23b784b4e6a2cc4"
server_key_iv: "f3a21ede6e4304dab4736fa85a452729"
server_key_version: "encryptionkey@0"
I20260620 09:26:36.978689 24417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781947562949448-16558-0/minicluster-data/ts-2/wal/instance:
uuid: "41d4efe5a7fd4b2a94dd1489ff6f3482"
format_stamp: "Formatted at 2026-06-20 09:26:36 on dist-test-slave-706q"
server_key: "b4977a760701c0bfe23b784b4e6a2cc4"
server_key_iv: "f3a21ede6e4304dab4736fa85a452729"
server_key_version: "encryptionkey@0"
I20260620 09:26:36.980178 24417 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 09:26:36.980906 24432 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:36.981113 24417 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 09:26:36.981184 24417 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781947562949448-16558-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781947562949448-16558-0/minicluster-data/ts-2/wal
uuid: "41d4efe5a7fd4b2a94dd1489ff6f3482"
format_stamp: "Formatted at 2026-06-20 09:26:36 on dist-test-slave-706q"
server_key: "b4977a760701c0bfe23b784b4e6a2cc4"
server_key_iv: "f3a21ede6e4304dab4736fa85a452729"
server_key_version: "encryptionkey@0"
I20260620 09:26:36.981236 24417 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781947562949448-16558-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781947562949448-16558-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781947562949448-16558-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:26:36.992568 24417 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:26:36.994912 24417 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:26:36.995062 24417 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:26:36.995512 24417 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:26:36.995608 24417 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:36.995658 24417 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:26:36.995684 24417 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:37.002636 24417 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.131:40739
I20260620 09:26:37.002743 24547 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.131:40739 every 8 connection(s)
I20260620 09:26:37.003053 24417 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781947562949448-16558-0/minicluster-data/ts-2/data/info.pb
Jun 20 09:26:37 dist-test-slave-706q krb5kdc[24020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947596, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.43.131@KRBTEST.COM for oryx/127.16.43.190@KRBTEST.COM
I20260620 09:26:37.004010 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 24417
I20260620 09:26:37.004142 16558 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781947562949448-16558-0/minicluster-data/ts-2/wal/instance
I20260620 09:26:37.004338 16558 external_mini_cluster.cc:1428] Setting key 9ebd505c2d2bea95c8115261644006ee
I20260620 09:26:37.007249 24125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:36.995993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.131:55055 (local address 127.16.43.190:36561)
0620 09:26:36.996041 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:36.996043 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:36.996377 (+ 334us) server_negotiation.cc:407] Connection header received
0620 09:26:36.997283 (+ 906us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:36.997284 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:36.997300 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:36.997378 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:36.998316 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:36.999280 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:37.000986 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:37.001240 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:37.001340 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:37.004287 (+ 2947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:37.004288 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:37.004289 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:26:37.004297 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:37.006024 (+ 1727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:37.006413 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:37.006413 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:37.006414 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:37.006469 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:37.006677 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:37.006678 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:37.006678 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:37.006822 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:37.006880 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:37.007083 (+ 203us) server_negotiation.cc:299] Negotiation successful
0620 09:26:37.007152 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 09:26:37.007332 24441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:36.996005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:36561 (local address 127.16.43.131:55055)
0620 09:26:36.996226 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:36.996264 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:36.997079 (+ 815us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:36.997506 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:36.997511 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:36.997917 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:36.998220 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:36.998226 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:36.999410 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:36.999411 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:37.000887 (+ 1476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:37.000890 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:37.001348 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:37.001349 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:37.001527 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:37.002224 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:26:37.002233 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:26:37.004161 (+ 1928us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:26:37.006102 (+ 1941us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:37.006108 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:37.006110 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:37.006326 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:37.006534 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:37.006534 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:37.006535 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:37.006622 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:37.006869 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:26:37.006876 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:26:37.006986 (+ 110us) client_negotiation.cc:769] Sending connection context
0620 09:26:37.007060 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 09:26:37.007178 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":81,"threads_started":1}
I20260620 09:26:37.007817 24548 heartbeater.cc:347] Connected to a master server at 127.16.43.190:36561
I20260620 09:26:37.007918 24548 heartbeater.cc:464] Registering TS with master...
I20260620 09:26:37.008100 24548 heartbeater.cc:511] Master 127.16.43.190:36561 requested a full tablet report, sending...
I20260620 09:26:37.008452 24065 ts_manager.cc:194] Registered new tserver with Master: 41d4efe5a7fd4b2a94dd1489ff6f3482 (127.16.43.131:40739)
I20260620 09:26:37.008849 24065 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.43.131@KRBTEST.COM'} at 127.16.43.131:55055
I20260620 09:26:37.014654 24125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:37.009001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.131:51101 (local address 127.16.43.190:36561)
0620 09:26:37.009041 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:37.009043 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:37.009142 (+ 99us) server_negotiation.cc:407] Connection header received
0620 09:26:37.009240 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:37.009241 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:37.009252 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:37.009308 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:37.009713 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:37.010361 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:37.011144 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:37.011339 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:37.011410 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:37.012341 (+ 931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:37.012343 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:37.012343 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:26:37.012349 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:37.013699 (+ 1350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:37.014000 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:37.014001 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:37.014001 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:37.014049 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:37.014206 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:37.014206 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:37.014207 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:37.014299 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:37.014351 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:37.014531 (+ 180us) server_negotiation.cc:299] Negotiation successful
0620 09:26:37.014589 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 09:26:37.014670 24554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:37.008954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:36561 (local address 127.16.43.131:51101)
0620 09:26:37.009084 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:37.009096 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:37.009185 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:37.009324 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:37.009325 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:37.009536 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:37.009591 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:37.009592 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:37.010460 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:37.010461 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:37.011072 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:37.011074 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:37.011428 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:37.011428 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:37.011476 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:37.011983 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:26:37.011988 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:26:37.012268 (+ 280us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:26:37.013778 (+ 1510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:37.013778 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:37.013779 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:37.013934 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:37.014106 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:37.014107 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:37.014107 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:37.014156 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:37.014385 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:26:37.014386 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:26:37.014432 (+ 46us) client_negotiation.cc:769] Sending connection context
0620 09:26:37.014507 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 09:26:37.014597 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":73,"threads_started":1}
I20260620 09:26:37.016530 16558 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 09:26:37 dist-test-slave-706q krb5kdc[24020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947597, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 09:26:37 dist-test-slave-706q krb5kdc[24020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.16.43.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:26:37 dist-test-slave-706q krb5kdc[24020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.16.43.190@KRBTEST.COM, Server not found in Kerberos database
W20260620 09:26:37.038043 24125 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:26:37.032587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48042 (local address 127.16.43.190:36561)
0620 09:26:37.032685 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:37.032687 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:37.032696 (+ 9us) server_negotiation.cc:407] Connection header received
0620 09:26:37.032724 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:37.032725 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:37.032736 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:37.032808 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:37.033371 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:37.033871 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:37.034610 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:37.034807 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:37.034893 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:37.037938 (+ 3045us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48042: BlockingRecv error: recv got EOF from 127.0.0.1:48042 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Jun 20 09:26:37 dist-test-slave-706q krb5kdc[24020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947597, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.16.43.190@KRBTEST.COM
I20260620 09:26:37.045912 24125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:37.038845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48052 (local address 127.16.43.190:36561)
0620 09:26:37.038903 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:37.038905 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:37.039026 (+ 121us) server_negotiation.cc:407] Connection header received
0620 09:26:37.039065 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:37.039065 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:37.039074 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:37.039123 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:37.039651 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:37.040118 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:37.041304 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:37.041543 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:37.041597 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:37.043735 (+ 2138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:37.043736 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:37.043736 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:26:37.043743 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:37.044932 (+ 1189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:37.045234 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:37.045235 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:37.045235 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:37.045286 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:37.045490 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:37.045490 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:37.045491 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:37.045589 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:37.045663 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:37.045743 (+ 80us) server_negotiation.cc:299] Negotiation successful
0620 09:26:37.045800 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 09:26:37.047811 24065 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48052:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 09:26:37.049417 24065 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 09:26:37.056071 24571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:37.051679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.130:35987 (local address 127.0.0.1:47812)
0620 09:26:37.051847 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:37.051900 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:37.051967 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:37.052450 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:37.052453 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:37.052456 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:26:37.052542 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:37.052543 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:37.053919 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:37.053920 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:37.054810 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:37.054813 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:37.055808 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:37.055809 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:37.055881 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:37.055886 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 09:26:37.055918 (+ 32us) client_negotiation.cc:240] Negotiation successful
0620 09:26:37.055980 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"mutex_wait_us":23,"thread_start_us":56,"threads_started":1}
I20260620 09:26:37.056177 24570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:37.051678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.131:40739 (local address 127.0.0.1:42064)
0620 09:26:37.052072 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:37.052081 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:37.052172 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:37.052462 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:37.052463 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:37.052464 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:26:37.052538 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:37.052539 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:37.053847 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:37.053848 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:37.054979 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:37.054981 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:37.055749 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:37.055750 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:37.055839 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:37.055862 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 09:26:37.055996 (+ 134us) client_negotiation.cc:240] Negotiation successful
0620 09:26:37.056044 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":381,"mutex_wait_us":29,"thread_start_us":77,"threads_started":1}
I20260620 09:26:37.056272 24572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:37.051687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.129:39361 (local address 127.0.0.1:51258)
0620 09:26:37.052253 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:37.052261 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:37.052305 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:37.052682 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:37.052683 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:37.052685 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:26:37.052727 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:37.052735 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:37.053571 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:37.053572 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:37.054891 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:37.054893 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:37.055895 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:37.055896 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:37.055947 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:37.055949 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 09:26:37.055995 (+ 46us) client_negotiation.cc:240] Negotiation successful
0620 09:26:37.056030 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"mutex_wait_us":45,"thread_start_us":82,"threads_started":1}
I20260620 09:26:37.056159 24574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:37.051844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42064 (local address 127.16.43.131:40739)
0620 09:26:37.052027 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:37.052032 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:37.052162 (+ 130us) server_negotiation.cc:407] Connection header received
0620 09:26:37.052253 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:37.052256 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:37.052278 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:37.052386 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:26:37.052629 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:37.053762 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:37.055054 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:37.055655 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:37.055748 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:37.055833 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:37.056014 (+ 181us) server_negotiation.cc:299] Negotiation successful
0620 09:26:37.056057 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":91,"threads_started":1}
I20260620 09:26:37.056528 24573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:37.051851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51258 (local address 127.16.43.129:39361)
0620 09:26:37.052027 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:37.052032 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:37.052369 (+ 337us) server_negotiation.cc:407] Connection header received
0620 09:26:37.052409 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:37.052411 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:37.052425 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:37.052565 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:26:37.052811 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:37.053482 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:37.054962 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:37.055706 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:37.055780 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:37.055853 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:37.056087 (+ 234us) server_negotiation.cc:299] Negotiation successful
0620 09:26:37.056431 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":73,"threads_started":1}
I20260620 09:26:37.056557 24575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:37.051924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47812 (local address 127.16.43.130:35987)
0620 09:26:37.052235 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:37.052239 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:37.052246 (+ 7us) server_negotiation.cc:407] Connection header received
0620 09:26:37.052275 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:37.052277 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:37.052288 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:37.052379 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:26:37.052629 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:37.053773 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:37.054910 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:37.055715 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:37.056291 (+ 576us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:37.056395 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:37.056422 (+ 27us) server_negotiation.cc:299] Negotiation successful
0620 09:26:37.056458 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":79,"threads_started":1}
I20260620 09:26:37.057255 24469 tablet_service.cc:1467] Processing CreateTablet for tablet cfc50a3d2ac94e19a99c07b472b66424 (DEFAULT_TABLE table=test-table [id=399d55941b1c4375b7f754bb01ef28bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:26:37.057252 24202 tablet_service.cc:1467] Processing CreateTablet for tablet cfc50a3d2ac94e19a99c07b472b66424 (DEFAULT_TABLE table=test-table [id=399d55941b1c4375b7f754bb01ef28bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:26:37.057580 24202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfc50a3d2ac94e19a99c07b472b66424. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:26:37.057592 24469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfc50a3d2ac94e19a99c07b472b66424. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:26:37.060652 24576 tablet_bootstrap.cc:492] T cfc50a3d2ac94e19a99c07b472b66424 P 6b68c749403d4f96a817a83a1a679a8d: Bootstrap starting.
I20260620 09:26:37.060680 24577 tablet_bootstrap.cc:492] T cfc50a3d2ac94e19a99c07b472b66424 P 41d4efe5a7fd4b2a94dd1489ff6f3482: Bootstrap starting.
I20260620 09:26:37.061149 24337 tablet_service.cc:1467] Processing CreateTablet for tablet cfc50a3d2ac94e19a99c07b472b66424 (DEFAULT_TABLE table=test-table [id=399d55941b1c4375b7f754bb01ef28bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:26:37.061316 24577 tablet_bootstrap.cc:654] T cfc50a3d2ac94e19a99c07b472b66424 P 41d4efe5a7fd4b2a94dd1489ff6f3482: Neither blocks nor log segments found. Creating new log.
I20260620 09:26:37.061403 24576 tablet_bootstrap.cc:654] T cfc50a3d2ac94e19a99c07b472b66424 P 6b68c749403d4f96a817a83a1a679a8d: Neither blocks nor log segments found. Creating new log.
I20260620 09:26:37.061532 24337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfc50a3d2ac94e19a99c07b472b66424. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:26:37.061662 24577 log.cc:824] T cfc50a3d2ac94e19a99c07b472b66424 P 41d4efe5a7fd4b2a94dd1489ff6f3482: Log is configured to *not* fsync() on all Append() calls
I20260620 09:26:37.061728 24576 log.cc:824] T cfc50a3d2ac94e19a99c07b472b66424 P 6b68c749403d4f96a817a83a1a679a8d: Log is configured to *not* fsync() on all Append() calls
I20260620 09:26:37.062366 24577 tablet_bootstrap.cc:492] T cfc50a3d2ac94e19a99c07b472b66424 P 41d4efe5a7fd4b2a94dd1489ff6f3482: No bootstrap required, opened a new log
I20260620 09:26:37.062458 24577 ts_tablet_manager.cc:1397] T cfc50a3d2ac94e19a99c07b472b66424 P 41d4efe5a7fd4b2a94dd1489ff6f3482: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 09:26:37.064172 24577 raft_consensus.cc:348] T cfc50a3d2ac94e19a99c07b472b66424 P 41d4efe5a7fd4b2a94dd1489ff6f3482 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b9304cb76a2417b956f9e63e78b7734" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 35987 } } peers { permanent_uuid: "41d4efe5a7fd4b2a94dd1489ff6f3482" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 40739 } } peers { permanent_uuid: "6b68c749403d4f96a817a83a1a679a8d" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 39361 } }
I20260620 09:26:37.064349 24577 raft_consensus.cc:374] T cfc50a3d2ac94e19a99c07b472b66424 P 41d4efe5a7fd4b2a94dd1489ff6f3482 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:26:37.064406 24577 raft_consensus.cc:729] T cfc50a3d2ac94e19a99c07b472b66424 P 41d4efe5a7fd4b2a94dd1489ff6f3482 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41d4efe5a7fd4b2a94dd1489ff6f3482, State: Initialized, Role: FOLLOWER
I20260620 09:26:37.064528 24577 consensus_queue.cc:260] T cfc50a3d2ac94e19a99c07b472b66424 P 41d4efe5a7fd4b2a94dd1489ff6f3482 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b9304cb76a2417b956f9e63e78b7734" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 35987 } } peers { permanent_uuid: "41d4efe5a7fd4b2a94dd1489ff6f3482" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 40739 } } peers { permanent_uuid: "6b68c749403d4f96a817a83a1a679a8d" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 39361 } }
I20260620 09:26:37.064841 24577 ts_tablet_manager.cc:1428] T cfc50a3d2ac94e19a99c07b472b66424 P 41d4efe5a7fd4b2a94dd1489ff6f3482: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 09:26:37.064884 24548 heartbeater.cc:503] Master 127.16.43.190:36561 was elected leader, sending a full tablet report...
I20260620 09:26:37.065043 24581 tablet_bootstrap.cc:492] T cfc50a3d2ac94e19a99c07b472b66424 P 6b9304cb76a2417b956f9e63e78b7734: Bootstrap starting.
I20260620 09:26:37.065770 24581 tablet_bootstrap.cc:654] T cfc50a3d2ac94e19a99c07b472b66424 P 6b9304cb76a2417b956f9e63e78b7734: Neither blocks nor log segments found. Creating new log.
I20260620 09:26:37.065783 24576 tablet_bootstrap.cc:492] T cfc50a3d2ac94e19a99c07b472b66424 P 6b68c749403d4f96a817a83a1a679a8d: No bootstrap required, opened a new log
I20260620 09:26:37.065864 24576 ts_tablet_manager.cc:1397] T cfc50a3d2ac94e19a99c07b472b66424 P 6b68c749403d4f96a817a83a1a679a8d: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260620 09:26:37.066126 24581 log.cc:824] T cfc50a3d2ac94e19a99c07b472b66424 P 6b9304cb76a2417b956f9e63e78b7734: Log is configured to *not* fsync() on all Append() calls
I20260620 09:26:37.066805 24581 tablet_bootstrap.cc:492] T cfc50a3d2ac94e19a99c07b472b66424 P 6b9304cb76a2417b956f9e63e78b7734: No bootstrap required, opened a new log
I20260620 09:26:37.066888 24581 ts_tablet_manager.cc:1397] T cfc50a3d2ac94e19a99c07b472b66424 P 6b9304cb76a2417b956f9e63e78b7734: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 09:26:37.067698 24576 raft_consensus.cc:348] T cfc50a3d2ac94e19a99c07b472b66424 P 6b68c749403d4f96a817a83a1a679a8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b9304cb76a2417b956f9e63e78b7734" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 35987 } } peers { permanent_uuid: "41d4efe5a7fd4b2a94dd1489ff6f3482" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 40739 } } peers { permanent_uuid: "6b68c749403d4f96a817a83a1a679a8d" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 39361 } }
I20260620 09:26:37.067879 24576 raft_consensus.cc:374] T cfc50a3d2ac94e19a99c07b472b66424 P 6b68c749403d4f96a817a83a1a679a8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:26:37.067929 24576 raft_consensus.cc:729] T cfc50a3d2ac94e19a99c07b472b66424 P 6b68c749403d4f96a817a83a1a679a8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b68c749403d4f96a817a83a1a679a8d, State: Initialized, Role: FOLLOWER
I20260620 09:26:37.068048 24576 consensus_queue.cc:260] T cfc50a3d2ac94e19a99c07b472b66424 P 6b68c749403d4f96a817a83a1a679a8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b9304cb76a2417b956f9e63e78b7734" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 35987 } } peers { permanent_uuid: "41d4efe5a7fd4b2a94dd1489ff6f3482" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 40739 } } peers { permanent_uuid: "6b68c749403d4f96a817a83a1a679a8d" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 39361 } }
I20260620 09:26:37.068317 24576 ts_tablet_manager.cc:1428] T cfc50a3d2ac94e19a99c07b472b66424 P 6b68c749403d4f96a817a83a1a679a8d: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260620 09:26:37.068397 24268 heartbeater.cc:503] Master 127.16.43.190:36561 was elected leader, sending a full tablet report...
I20260620 09:26:37.068837 24581 raft_consensus.cc:348] T cfc50a3d2ac94e19a99c07b472b66424 P 6b9304cb76a2417b956f9e63e78b7734 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b9304cb76a2417b956f9e63e78b7734" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 35987 } } peers { permanent_uuid: "41d4efe5a7fd4b2a94dd1489ff6f3482" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 40739 } } peers { permanent_uuid: "6b68c749403d4f96a817a83a1a679a8d" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 39361 } }
I20260620 09:26:37.068984 24581 raft_consensus.cc:374] T cfc50a3d2ac94e19a99c07b472b66424 P 6b9304cb76a2417b956f9e63e78b7734 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:26:37.069026 24581 raft_consensus.cc:729] T cfc50a3d2ac94e19a99c07b472b66424 P 6b9304cb76a2417b956f9e63e78b7734 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b9304cb76a2417b956f9e63e78b7734, State: Initialized, Role: FOLLOWER
I20260620 09:26:37.069144 24581 consensus_queue.cc:260] T cfc50a3d2ac94e19a99c07b472b66424 P 6b9304cb76a2417b956f9e63e78b7734 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b9304cb76a2417b956f9e63e78b7734" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 35987 } } peers { permanent_uuid: "41d4efe5a7fd4b2a94dd1489ff6f3482" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 40739 } } peers { permanent_uuid: "6b68c749403d4f96a817a83a1a679a8d" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 39361 } }
I20260620 09:26:37.069415 24581 ts_tablet_manager.cc:1428] T cfc50a3d2ac94e19a99c07b472b66424 P 6b9304cb76a2417b956f9e63e78b7734: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 09:26:37.069440 24408 heartbeater.cc:503] Master 127.16.43.190:36561 was elected leader, sending a full tablet report...
W20260620 09:26:37.102221 24409 tablet.cc:2369] T cfc50a3d2ac94e19a99c07b472b66424 P 6b9304cb76a2417b956f9e63e78b7734: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 09:26:37.193112 24269 tablet.cc:2369] T cfc50a3d2ac94e19a99c07b472b66424 P 6b68c749403d4f96a817a83a1a679a8d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 09:26:37.253713 24549 tablet.cc:2369] T cfc50a3d2ac94e19a99c07b472b66424 P 41d4efe5a7fd4b2a94dd1489ff6f3482: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 09:26:37.276938 24583 raft_consensus.cc:482] T cfc50a3d2ac94e19a99c07b472b66424 P 6b68c749403d4f96a817a83a1a679a8d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:26:37.277074 24583 raft_consensus.cc:504] T cfc50a3d2ac94e19a99c07b472b66424 P 6b68c749403d4f96a817a83a1a679a8d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b9304cb76a2417b956f9e63e78b7734" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 35987 } } peers { permanent_uuid: "41d4efe5a7fd4b2a94dd1489ff6f3482" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 40739 } } peers { permanent_uuid: "6b68c749403d4f96a817a83a1a679a8d" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 39361 } }
I20260620 09:26:37.277452 24583 leader_election.cc:288] T cfc50a3d2ac94e19a99c07b472b66424 P 6b68c749403d4f96a817a83a1a679a8d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6b9304cb76a2417b956f9e63e78b7734 (127.16.43.130:35987), 41d4efe5a7fd4b2a94dd1489ff6f3482 (127.16.43.131:40739)
I20260620 09:26:37.280709 24574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:37.277834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.129:49887 (local address 127.16.43.131:40739)
0620 09:26:37.277894 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:37.277896 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:37.277906 (+ 10us) server_negotiation.cc:407] Connection header received
0620 09:26:37.278032 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:37.278033 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:37.278043 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:37.278074 (+ 31us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:26:37.278299 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:37.278774 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:37.279962 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:37.280340 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:37.280426 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:37.280435 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:37.280586 (+ 151us) server_negotiation.cc:299] Negotiation successful
0620 09:26:37.280640 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 09:26:37.280720 24585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:37.277674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.131:40739 (local address 127.16.43.129:49887)
0620 09:26:37.277834 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:37.277846 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:37.277907 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:37.278152 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:37.278153 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:37.278155 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:26:37.278219 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:37.278226 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:37.278924 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:37.278925 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:37.279876 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:37.279879 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:37.280426 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:37.280427 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:37.280498 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:37.280502 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 09:26:37.280577 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 09:26:37.280623 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"mutex_wait_us":45,"thread_start_us":73,"threads_started":1}
I20260620 09:26:37.280771 24575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:37.277855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.129:50857 (local address 127.16.43.130:35987)
0620 09:26:37.277945 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:37.277947 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:37.278027 (+ 80us) server_negotiation.cc:407] Connection header received
0620 09:26:37.278062 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:37.278063 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:37.278073 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:37.278131 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:26:37.278334 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:37.278806 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:37.279953 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:37.280414 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:37.280498 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:37.280506 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:37.280657 (+ 151us) server_negotiation.cc:299] Negotiation successful
0620 09:26:37.280694 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 09:26:37.280826 24586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:37.277676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.130:35987 (local address 127.16.43.129:50857)
0620 09:26:37.277950 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:37.277959 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:37.278015 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:37.278149 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:37.278150 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:37.278153 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:26:37.278236 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:37.278238 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:37.278937 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:37.278939 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:37.279854 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:37.279856 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:37.280498 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:37.280499 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:37.280578 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:37.280581 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 09:26:37.280650 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 09:26:37.280691 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"mutex_wait_us":16,"thread_start_us":70,"threads_started":1}
I20260620 09:26:37.281141 24362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfc50a3d2ac94e19a99c07b472b66424" candidate_uuid: "6b68c749403d4f96a817a83a1a679a8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b9304cb76a2417b956f9e63e78b7734" is_pre_election: true
I20260620 09:26:37.281118 24502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfc50a3d2ac94e19a99c07b472b66424" candidate_uuid: "6b68c749403d4f96a817a83a1a679a8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41d4efe5a7fd4b2a94dd1489ff6f3482" is_pre_election: true
I20260620 09:26:37.281304 24502 raft_consensus.cc:2454] T cfc50a3d2ac94e19a99c07b472b66424 P 41d4efe5a7fd4b2a94dd1489ff6f3482 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b68c749403d4f96a817a83a1a679a8d in term 0.
I20260620 09:26:37.281309 24362 raft_consensus.cc:2454] T cfc50a3d2ac94e19a99c07b472b66424 P 6b9304cb76a2417b956f9e63e78b7734 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b68c749403d4f96a817a83a1a679a8d in term 0.
I20260620 09:26:37.281564 24153 leader_election.cc:302] T cfc50a3d2ac94e19a99c07b472b66424 P 6b68c749403d4f96a817a83a1a679a8d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41d4efe5a7fd4b2a94dd1489ff6f3482, 6b68c749403d4f96a817a83a1a679a8d; no voters:
I20260620 09:26:37.281706 24583 raft_consensus.cc:2781] T cfc50a3d2ac94e19a99c07b472b66424 P 6b68c749403d4f96a817a83a1a679a8d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 09:26:37.281765 24583 raft_consensus.cc:482] T cfc50a3d2ac94e19a99c07b472b66424 P 6b68c749403d4f96a817a83a1a679a8d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 09:26:37.281790 24583 raft_consensus.cc:3037] T cfc50a3d2ac94e19a99c07b472b66424 P 6b68c749403d4f96a817a83a1a679a8d [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:26:37.282461 24583 raft_consensus.cc:504] T cfc50a3d2ac94e19a99c07b472b66424 P 6b68c749403d4f96a817a83a1a679a8d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b9304cb76a2417b956f9e63e78b7734" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 35987 } } peers { permanent_uuid: "41d4efe5a7fd4b2a94dd1489ff6f3482" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 40739 } } peers { permanent_uuid: "6b68c749403d4f96a817a83a1a679a8d" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 39361 } }
I20260620 09:26:37.282608 24583 leader_election.cc:288] T cfc50a3d2ac94e19a99c07b472b66424 P 6b68c749403d4f96a817a83a1a679a8d [CANDIDATE]: Term 1 election: Requested vote from peers 6b9304cb76a2417b956f9e63e78b7734 (127.16.43.130:35987), 41d4efe5a7fd4b2a94dd1489ff6f3482 (127.16.43.131:40739)
I20260620 09:26:37.282773 24502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfc50a3d2ac94e19a99c07b472b66424" candidate_uuid: "6b68c749403d4f96a817a83a1a679a8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41d4efe5a7fd4b2a94dd1489ff6f3482"
I20260620 09:26:37.282773 24362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfc50a3d2ac94e19a99c07b472b66424" candidate_uuid: "6b68c749403d4f96a817a83a1a679a8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b9304cb76a2417b956f9e63e78b7734"
I20260620 09:26:37.282871 24502 raft_consensus.cc:3037] T cfc50a3d2ac94e19a99c07b472b66424 P 41d4efe5a7fd4b2a94dd1489ff6f3482 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:26:37.282922 24362 raft_consensus.cc:3037] T cfc50a3d2ac94e19a99c07b472b66424 P 6b9304cb76a2417b956f9e63e78b7734 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:26:37.283699 24502 raft_consensus.cc:2454] T cfc50a3d2ac94e19a99c07b472b66424 P 41d4efe5a7fd4b2a94dd1489ff6f3482 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b68c749403d4f96a817a83a1a679a8d in term 1.
I20260620 09:26:37.283710 24362 raft_consensus.cc:2454] T cfc50a3d2ac94e19a99c07b472b66424 P 6b9304cb76a2417b956f9e63e78b7734 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b68c749403d4f96a817a83a1a679a8d in term 1.
I20260620 09:26:37.283880 24154 leader_election.cc:302] T cfc50a3d2ac94e19a99c07b472b66424 P 6b68c749403d4f96a817a83a1a679a8d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b68c749403d4f96a817a83a1a679a8d, 6b9304cb76a2417b956f9e63e78b7734; no voters:
I20260620 09:26:37.284018 24583 raft_consensus.cc:2781] T cfc50a3d2ac94e19a99c07b472b66424 P 6b68c749403d4f96a817a83a1a679a8d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:26:37.284149 24583 raft_consensus.cc:686] T cfc50a3d2ac94e19a99c07b472b66424 P 6b68c749403d4f96a817a83a1a679a8d [term 1 LEADER]: Becoming Leader. State: Replica: 6b68c749403d4f96a817a83a1a679a8d, State: Running, Role: LEADER
I20260620 09:26:37.284252 24583 consensus_queue.cc:237] T cfc50a3d2ac94e19a99c07b472b66424 P 6b68c749403d4f96a817a83a1a679a8d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b9304cb76a2417b956f9e63e78b7734" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 35987 } } peers { permanent_uuid: "41d4efe5a7fd4b2a94dd1489ff6f3482" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 40739 } } peers { permanent_uuid: "6b68c749403d4f96a817a83a1a679a8d" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 39361 } }
I20260620 09:26:37.285131 24065 catalog_manager.cc:5495] T cfc50a3d2ac94e19a99c07b472b66424 P 6b68c749403d4f96a817a83a1a679a8d reported cstate change: term changed from 0 to 1, leader changed from <none> to 6b68c749403d4f96a817a83a1a679a8d (127.16.43.129). New cstate: current_term: 1 leader_uuid: "6b68c749403d4f96a817a83a1a679a8d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b9304cb76a2417b956f9e63e78b7734" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 35987 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "41d4efe5a7fd4b2a94dd1489ff6f3482" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 40739 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b68c749403d4f96a817a83a1a679a8d" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 39361 } health_report { overall_health: HEALTHY } } }
I20260620 09:26:37.339170 24573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:37.336605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51266 (local address 127.16.43.129:39361)
0620 09:26:37.336678 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:37.336680 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:37.336690 (+ 10us) server_negotiation.cc:407] Connection header received
0620 09:26:37.336735 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:37.336736 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:37.336755 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:37.336836 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:26:37.336980 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:37.337509 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:37.338261 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:37.338512 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:37.338584 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:37.338802 (+ 218us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:26:37.338930 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:26:37.339026 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:37.339046 (+ 20us) server_negotiation.cc:299] Negotiation successful
0620 09:26:37.339067 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 09:26:37.341105 24502 raft_consensus.cc:1261] T cfc50a3d2ac94e19a99c07b472b66424 P 41d4efe5a7fd4b2a94dd1489ff6f3482 [term 1 FOLLOWER]: Refusing update from remote peer 6b68c749403d4f96a817a83a1a679a8d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:26:37.341130 24362 raft_consensus.cc:1261] T cfc50a3d2ac94e19a99c07b472b66424 P 6b9304cb76a2417b956f9e63e78b7734 [term 1 FOLLOWER]: Refusing update from remote peer 6b68c749403d4f96a817a83a1a679a8d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:26:37.341511 24587 consensus_queue.cc:1035] T cfc50a3d2ac94e19a99c07b472b66424 P 6b68c749403d4f96a817a83a1a679a8d [LEADER]: Connected to new peer: Peer: permanent_uuid: "41d4efe5a7fd4b2a94dd1489ff6f3482" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 40739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:26:37.341646 24583 consensus_queue.cc:1035] T cfc50a3d2ac94e19a99c07b472b66424 P 6b68c749403d4f96a817a83a1a679a8d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9304cb76a2417b956f9e63e78b7734" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 35987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:26:37.345638 24590 mvcc.cc:205] Tried to move back new op lower bound from 7298857358706442240 to 7298857358476902400. Current Snapshot: MvccSnapshot[applied={T|T < 7298857358706442240 or (T in {7298857358706442240})}]
I20260620 09:26:37.347090 24064 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48052:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:26:37.347193 24064 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48052:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:26:37.348031 24064 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d0f24a0b0e524fd884b6522ce4569de5: Sending DeleteTablet for 3 replicas of tablet cfc50a3d2ac94e19a99c07b472b66424
I20260620 09:26:37.348381 24469 tablet_service.cc:1514] Processing DeleteTablet for tablet cfc50a3d2ac94e19a99c07b472b66424 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:26:37 UTC) from {username='oryx', principal='oryx/127.16.43.190@KRBTEST.COM'} at 127.0.0.1:42064
I20260620 09:26:37.348425 24202 tablet_service.cc:1514] Processing DeleteTablet for tablet cfc50a3d2ac94e19a99c07b472b66424 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:26:37 UTC) from {username='oryx', principal='oryx/127.16.43.190@KRBTEST.COM'} at 127.0.0.1:51258
I20260620 09:26:37.348466 24337 tablet_service.cc:1514] Processing DeleteTablet for tablet cfc50a3d2ac94e19a99c07b472b66424 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:26:37 UTC) from {username='oryx', principal='oryx/127.16.43.190@KRBTEST.COM'} at 127.0.0.1:47812
I20260620 09:26:37.348562 24064 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48052:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 09:26:37.348706 24064 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 09:26:37.348932 24600 tablet_replica.cc:330] T cfc50a3d2ac94e19a99c07b472b66424 P 41d4efe5a7fd4b2a94dd1489ff6f3482: stopping tablet replica
I20260620 09:26:37.349045 24600 raft_consensus.cc:2229] T cfc50a3d2ac94e19a99c07b472b66424 P 41d4efe5a7fd4b2a94dd1489ff6f3482 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:26:37.349188 24600 raft_consensus.cc:2258] T cfc50a3d2ac94e19a99c07b472b66424 P 41d4efe5a7fd4b2a94dd1489ff6f3482 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:26:37.349925 24202 tablet_service.cc:1467] Processing CreateTablet for tablet be4756bd86944a7f954f515ecea00d40 (DEFAULT_TABLE table=test-table [id=8d1f3ae5c68b4cf0941178e1fc50f5bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:26:37.349998 24337 tablet_service.cc:1467] Processing CreateTablet for tablet be4756bd86944a7f954f515ecea00d40 (DEFAULT_TABLE table=test-table [id=8d1f3ae5c68b4cf0941178e1fc50f5bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:26:37.350059 24202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be4756bd86944a7f954f515ecea00d40. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:26:37.350127 24337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be4756bd86944a7f954f515ecea00d40. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:26:37.350244 24469 tablet_service.cc:1467] Processing CreateTablet for tablet be4756bd86944a7f954f515ecea00d40 (DEFAULT_TABLE table=test-table [id=8d1f3ae5c68b4cf0941178e1fc50f5bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:26:37.350342 24469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be4756bd86944a7f954f515ecea00d40. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:26:37.351459 24576 tablet_bootstrap.cc:492] T be4756bd86944a7f954f515ecea00d40 P 6b68c749403d4f96a817a83a1a679a8d: Bootstrap starting.
I20260620 09:26:37.351548 24581 tablet_bootstrap.cc:492] T be4756bd86944a7f954f515ecea00d40 P 6b9304cb76a2417b956f9e63e78b7734: Bootstrap starting.
I20260620 09:26:37.351913 24576 tablet_bootstrap.cc:654] T be4756bd86944a7f954f515ecea00d40 P 6b68c749403d4f96a817a83a1a679a8d: Neither blocks nor log segments found. Creating new log.
I20260620 09:26:37.352125 24581 tablet_bootstrap.cc:654] T be4756bd86944a7f954f515ecea00d40 P 6b9304cb76a2417b956f9e63e78b7734: Neither blocks nor log segments found. Creating new log.
I20260620 09:26:37.352461 24577 tablet_bootstrap.cc:492] T be4756bd86944a7f954f515ecea00d40 P 41d4efe5a7fd4b2a94dd1489ff6f3482: Bootstrap starting.
I20260620 09:26:37.352628 24576 tablet_bootstrap.cc:492] T be4756bd86944a7f954f515ecea00d40 P 6b68c749403d4f96a817a83a1a679a8d: No bootstrap required, opened a new log
I20260620 09:26:37.352669 24576 ts_tablet_manager.cc:1397] T be4756bd86944a7f954f515ecea00d40 P 6b68c749403d4f96a817a83a1a679a8d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 09:26:37.352806 24577 tablet_bootstrap.cc:654] T be4756bd86944a7f954f515ecea00d40 P 41d4efe5a7fd4b2a94dd1489ff6f3482: Neither blocks nor log segments found. Creating new log.
I20260620 09:26:37.352811 24576 raft_consensus.cc:348] T be4756bd86944a7f954f515ecea00d40 P 6b68c749403d4f96a817a83a1a679a8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b68c749403d4f96a817a83a1a679a8d" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 39361 } } peers { permanent_uuid: "6b9304cb76a2417b956f9e63e78b7734" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 35987 } } peers { permanent_uuid: "41d4efe5a7fd4b2a94dd1489ff6f3482" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 40739 } }
I20260620 09:26:37.352902 24576 raft_consensus.cc:374] T be4756bd86944a7f954f515ecea00d40 P 6b68c749403d4f96a817a83a1a679a8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:26:37.352928 24576 raft_consensus.cc:729] T be4756bd86944a7f954f515ecea00d40 P 6b68c749403d4f96a817a83a1a679a8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b68c749403d4f96a817a83a1a679a8d, State: Initialized, Role: FOLLOWER
I20260620 09:26:37.352983 24576 consensus_queue.cc:260] T be4756bd86944a7f954f515ecea00d40 P 6b68c749403d4f96a817a83a1a679a8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b68c749403d4f96a817a83a1a679a8d" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 39361 } } peers { permanent_uuid: "6b9304cb76a2417b956f9e63e78b7734" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 35987 } } peers { permanent_uuid: "41d4efe5a7fd4b2a94dd1489ff6f3482" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 40739 } }
I20260620 09:26:37.353065 24576 ts_tablet_manager.cc:1428] T be4756bd86944a7f954f515ecea00d40 P 6b68c749403d4f96a817a83a1a679a8d: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 09:26:37.353643 24577 tablet_bootstrap.cc:492] T be4756bd86944a7f954f515ecea00d40 P 41d4efe5a7fd4b2a94dd1489ff6f3482: No bootstrap required, opened a new log
I20260620 09:26:37.353685 24577 ts_tablet_manager.cc:1397] T be4756bd86944a7f954f515ecea00d40 P 41d4efe5a7fd4b2a94dd1489ff6f3482: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 09:26:37.353794 24577 raft_consensus.cc:348] T be4756bd86944a7f954f515ecea00d40 P 41d4efe5a7fd4b2a94dd1489ff6f3482 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b68c749403d4f96a817a83a1a679a8d" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 39361 } } peers { permanent_uuid: "6b9304cb76a2417b956f9e63e78b7734" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 35987 } } peers { permanent_uuid: "41d4efe5a7fd4b2a94dd1489ff6f3482" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 40739 } }
I20260620 09:26:37.353848 24577 raft_consensus.cc:374] T be4756bd86944a7f954f515ecea00d40 P 41d4efe5a7fd4b2a94dd1489ff6f3482 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:26:37.353861 24577 raft_consensus.cc:729] T be4756bd86944a7f954f515ecea00d40 P 41d4efe5a7fd4b2a94dd1489ff6f3482 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41d4efe5a7fd4b2a94dd1489ff6f3482, State: Initialized, Role: FOLLOWER
I20260620 09:26:37.353919 24577 consensus_queue.cc:260] T be4756bd86944a7f954f515ecea00d40 P 41d4efe5a7fd4b2a94dd1489ff6f3482 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b68c749403d4f96a817a83a1a679a8d" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 39361 } } peers { permanent_uuid: "6b9304cb76a2417b956f9e63e78b7734" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 35987 } } peers { permanent_uuid: "41d4efe5a7fd4b2a94dd1489ff6f3482" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 40739 } }
I20260620 09:26:37.354012 24577 ts_tablet_manager.cc:1428] T be4756bd86944a7f954f515ecea00d40 P 41d4efe5a7fd4b2a94dd1489ff6f3482: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:37.354692 24581 tablet_bootstrap.cc:492] T be4756bd86944a7f954f515ecea00d40 P 6b9304cb76a2417b956f9e63e78b7734: No bootstrap required, opened a new log
I20260620 09:26:37.354744 24581 ts_tablet_manager.cc:1397] T be4756bd86944a7f954f515ecea00d40 P 6b9304cb76a2417b956f9e63e78b7734: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260620 09:26:37.354918 24600 ts_tablet_manager.cc:1905] T cfc50a3d2ac94e19a99c07b472b66424 P 41d4efe5a7fd4b2a94dd1489ff6f3482: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:26:37.354897 24581 raft_consensus.cc:348] T be4756bd86944a7f954f515ecea00d40 P 6b9304cb76a2417b956f9e63e78b7734 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b68c749403d4f96a817a83a1a679a8d" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 39361 } } peers { permanent_uuid: "6b9304cb76a2417b956f9e63e78b7734" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 35987 } } peers { permanent_uuid: "41d4efe5a7fd4b2a94dd1489ff6f3482" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 40739 } }
I20260620 09:26:37.355001 24581 raft_consensus.cc:374] T be4756bd86944a7f954f515ecea00d40 P 6b9304cb76a2417b956f9e63e78b7734 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:26:37.355029 24581 raft_consensus.cc:729] T be4756bd86944a7f954f515ecea00d40 P 6b9304cb76a2417b956f9e63e78b7734 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b9304cb76a2417b956f9e63e78b7734, State: Initialized, Role: FOLLOWER
I20260620 09:26:37.355074 24581 consensus_queue.cc:260] T be4756bd86944a7f954f515ecea00d40 P 6b9304cb76a2417b956f9e63e78b7734 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b68c749403d4f96a817a83a1a679a8d" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 39361 } } peers { permanent_uuid: "6b9304cb76a2417b956f9e63e78b7734" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 35987 } } peers { permanent_uuid: "41d4efe5a7fd4b2a94dd1489ff6f3482" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 40739 } }
I20260620 09:26:37.355168 24581 ts_tablet_manager.cc:1428] T be4756bd86944a7f954f515ecea00d40 P 6b9304cb76a2417b956f9e63e78b7734: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:37.355852 24601 tablet_replica.cc:330] T cfc50a3d2ac94e19a99c07b472b66424 P 6b68c749403d4f96a817a83a1a679a8d: stopping tablet replica
I20260620 09:26:37.355973 24602 tablet_replica.cc:330] T cfc50a3d2ac94e19a99c07b472b66424 P 6b9304cb76a2417b956f9e63e78b7734: stopping tablet replica
I20260620 09:26:37.356109 24602 raft_consensus.cc:2229] T cfc50a3d2ac94e19a99c07b472b66424 P 6b9304cb76a2417b956f9e63e78b7734 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:26:37.356237 24602 raft_consensus.cc:2258] T cfc50a3d2ac94e19a99c07b472b66424 P 6b9304cb76a2417b956f9e63e78b7734 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:26:37.356439 24601 raft_consensus.cc:2229] T cfc50a3d2ac94e19a99c07b472b66424 P 6b68c749403d4f96a817a83a1a679a8d [term 1 LEADER]: Raft consensus shutting down.
I20260620 09:26:37.356604 24601 raft_consensus.cc:2258] T cfc50a3d2ac94e19a99c07b472b66424 P 6b68c749403d4f96a817a83a1a679a8d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:26:37.356618 24600 ts_tablet_manager.cc:1918] T cfc50a3d2ac94e19a99c07b472b66424 P 41d4efe5a7fd4b2a94dd1489ff6f3482: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:26:37.356686 24600 log.cc:1196] T cfc50a3d2ac94e19a99c07b472b66424 P 41d4efe5a7fd4b2a94dd1489ff6f3482: Deleting WAL directory at /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781947562949448-16558-0/minicluster-data/ts-2/wal/wals/cfc50a3d2ac94e19a99c07b472b66424
I20260620 09:26:37.357013 24600 ts_tablet_manager.cc:1939] T cfc50a3d2ac94e19a99c07b472b66424 P 41d4efe5a7fd4b2a94dd1489ff6f3482: Deleting consensus metadata
I20260620 09:26:37.357070 24602 ts_tablet_manager.cc:1905] T cfc50a3d2ac94e19a99c07b472b66424 P 6b9304cb76a2417b956f9e63e78b7734: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:26:37.357070 24601 ts_tablet_manager.cc:1905] T cfc50a3d2ac94e19a99c07b472b66424 P 6b68c749403d4f96a817a83a1a679a8d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:26:37.357517 24049 catalog_manager.cc:4841] TS 41d4efe5a7fd4b2a94dd1489ff6f3482 (127.16.43.131:40739): tablet cfc50a3d2ac94e19a99c07b472b66424 (table test-table [id=399d55941b1c4375b7f754bb01ef28bf]) successfully deleted
I20260620 09:26:37.358765 24601 ts_tablet_manager.cc:1918] T cfc50a3d2ac94e19a99c07b472b66424 P 6b68c749403d4f96a817a83a1a679a8d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:26:37.358848 24601 log.cc:1196] T cfc50a3d2ac94e19a99c07b472b66424 P 6b68c749403d4f96a817a83a1a679a8d: Deleting WAL directory at /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781947562949448-16558-0/minicluster-data/ts-0/wal/wals/cfc50a3d2ac94e19a99c07b472b66424
I20260620 09:26:37.359102 24601 ts_tablet_manager.cc:1939] T cfc50a3d2ac94e19a99c07b472b66424 P 6b68c749403d4f96a817a83a1a679a8d: Deleting consensus metadata
I20260620 09:26:37.359205 24602 ts_tablet_manager.cc:1918] T cfc50a3d2ac94e19a99c07b472b66424 P 6b9304cb76a2417b956f9e63e78b7734: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:26:37.359277 24602 log.cc:1196] T cfc50a3d2ac94e19a99c07b472b66424 P 6b9304cb76a2417b956f9e63e78b7734: Deleting WAL directory at /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781947562949448-16558-0/minicluster-data/ts-1/wal/wals/cfc50a3d2ac94e19a99c07b472b66424
I20260620 09:26:37.359555 24602 ts_tablet_manager.cc:1939] T cfc50a3d2ac94e19a99c07b472b66424 P 6b9304cb76a2417b956f9e63e78b7734: Deleting consensus metadata
I20260620 09:26:37.359568 24052 catalog_manager.cc:4841] TS 6b68c749403d4f96a817a83a1a679a8d (127.16.43.129:39361): tablet cfc50a3d2ac94e19a99c07b472b66424 (table test-table [id=399d55941b1c4375b7f754bb01ef28bf]) successfully deleted
I20260620 09:26:37.360113 24050 catalog_manager.cc:4841] TS 6b9304cb76a2417b956f9e63e78b7734 (127.16.43.130:35987): tablet cfc50a3d2ac94e19a99c07b472b66424 (table test-table [id=399d55941b1c4375b7f754bb01ef28bf]) successfully deleted
I20260620 09:26:37.478993 24583 raft_consensus.cc:482] T be4756bd86944a7f954f515ecea00d40 P 6b68c749403d4f96a817a83a1a679a8d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:26:37.479187 24583 raft_consensus.cc:504] T be4756bd86944a7f954f515ecea00d40 P 6b68c749403d4f96a817a83a1a679a8d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b68c749403d4f96a817a83a1a679a8d" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 39361 } } peers { permanent_uuid: "6b9304cb76a2417b956f9e63e78b7734" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 35987 } } peers { permanent_uuid: "41d4efe5a7fd4b2a94dd1489ff6f3482" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 40739 } }
I20260620 09:26:37.479413 24583 leader_election.cc:288] T be4756bd86944a7f954f515ecea00d40 P 6b68c749403d4f96a817a83a1a679a8d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6b9304cb76a2417b956f9e63e78b7734 (127.16.43.130:35987), 41d4efe5a7fd4b2a94dd1489ff6f3482 (127.16.43.131:40739)
I20260620 09:26:37.479698 24502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be4756bd86944a7f954f515ecea00d40" candidate_uuid: "6b68c749403d4f96a817a83a1a679a8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41d4efe5a7fd4b2a94dd1489ff6f3482" is_pre_election: true
I20260620 09:26:37.479696 24362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be4756bd86944a7f954f515ecea00d40" candidate_uuid: "6b68c749403d4f96a817a83a1a679a8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b9304cb76a2417b956f9e63e78b7734" is_pre_election: true
I20260620 09:26:37.479823 24502 raft_consensus.cc:2454] T be4756bd86944a7f954f515ecea00d40 P 41d4efe5a7fd4b2a94dd1489ff6f3482 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b68c749403d4f96a817a83a1a679a8d in term 0.
I20260620 09:26:37.479828 24362 raft_consensus.cc:2454] T be4756bd86944a7f954f515ecea00d40 P 6b9304cb76a2417b956f9e63e78b7734 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b68c749403d4f96a817a83a1a679a8d in term 0.
I20260620 09:26:37.480022 24153 leader_election.cc:302] T be4756bd86944a7f954f515ecea00d40 P 6b68c749403d4f96a817a83a1a679a8d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41d4efe5a7fd4b2a94dd1489ff6f3482, 6b68c749403d4f96a817a83a1a679a8d; no voters:
I20260620 09:26:37.480139 24583 raft_consensus.cc:2781] T be4756bd86944a7f954f515ecea00d40 P 6b68c749403d4f96a817a83a1a679a8d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 09:26:37.480180 24583 raft_consensus.cc:482] T be4756bd86944a7f954f515ecea00d40 P 6b68c749403d4f96a817a83a1a679a8d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 09:26:37.480206 24583 raft_consensus.cc:3037] T be4756bd86944a7f954f515ecea00d40 P 6b68c749403d4f96a817a83a1a679a8d [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:26:37.480962 24583 raft_consensus.cc:504] T be4756bd86944a7f954f515ecea00d40 P 6b68c749403d4f96a817a83a1a679a8d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b68c749403d4f96a817a83a1a679a8d" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 39361 } } peers { permanent_uuid: "6b9304cb76a2417b956f9e63e78b7734" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 35987 } } peers { permanent_uuid: "41d4efe5a7fd4b2a94dd1489ff6f3482" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 40739 } }
I20260620 09:26:37.481112 24583 leader_election.cc:288] T be4756bd86944a7f954f515ecea00d40 P 6b68c749403d4f96a817a83a1a679a8d [CANDIDATE]: Term 1 election: Requested vote from peers 6b9304cb76a2417b956f9e63e78b7734 (127.16.43.130:35987), 41d4efe5a7fd4b2a94dd1489ff6f3482 (127.16.43.131:40739)
I20260620 09:26:37.481220 24362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be4756bd86944a7f954f515ecea00d40" candidate_uuid: "6b68c749403d4f96a817a83a1a679a8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b9304cb76a2417b956f9e63e78b7734"
I20260620 09:26:37.481269 24502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be4756bd86944a7f954f515ecea00d40" candidate_uuid: "6b68c749403d4f96a817a83a1a679a8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41d4efe5a7fd4b2a94dd1489ff6f3482"
I20260620 09:26:37.481308 24362 raft_consensus.cc:3037] T be4756bd86944a7f954f515ecea00d40 P 6b9304cb76a2417b956f9e63e78b7734 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:26:37.481339 24502 raft_consensus.cc:3037] T be4756bd86944a7f954f515ecea00d40 P 41d4efe5a7fd4b2a94dd1489ff6f3482 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:26:37.481936 24362 raft_consensus.cc:2454] T be4756bd86944a7f954f515ecea00d40 P 6b9304cb76a2417b956f9e63e78b7734 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b68c749403d4f96a817a83a1a679a8d in term 1.
I20260620 09:26:37.481945 24502 raft_consensus.cc:2454] T be4756bd86944a7f954f515ecea00d40 P 41d4efe5a7fd4b2a94dd1489ff6f3482 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b68c749403d4f96a817a83a1a679a8d in term 1.
I20260620 09:26:37.482115 24154 leader_election.cc:302] T be4756bd86944a7f954f515ecea00d40 P 6b68c749403d4f96a817a83a1a679a8d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b68c749403d4f96a817a83a1a679a8d, 6b9304cb76a2417b956f9e63e78b7734; no voters:
I20260620 09:26:37.482226 24583 raft_consensus.cc:2781] T be4756bd86944a7f954f515ecea00d40 P 6b68c749403d4f96a817a83a1a679a8d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:26:37.482280 24583 raft_consensus.cc:686] T be4756bd86944a7f954f515ecea00d40 P 6b68c749403d4f96a817a83a1a679a8d [term 1 LEADER]: Becoming Leader. State: Replica: 6b68c749403d4f96a817a83a1a679a8d, State: Running, Role: LEADER
I20260620 09:26:37.482353 24583 consensus_queue.cc:237] T be4756bd86944a7f954f515ecea00d40 P 6b68c749403d4f96a817a83a1a679a8d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b68c749403d4f96a817a83a1a679a8d" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 39361 } } peers { permanent_uuid: "6b9304cb76a2417b956f9e63e78b7734" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 35987 } } peers { permanent_uuid: "41d4efe5a7fd4b2a94dd1489ff6f3482" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 40739 } }
I20260620 09:26:37.482985 24064 catalog_manager.cc:5495] T be4756bd86944a7f954f515ecea00d40 P 6b68c749403d4f96a817a83a1a679a8d reported cstate change: term changed from 0 to 1, leader changed from <none> to 6b68c749403d4f96a817a83a1a679a8d (127.16.43.129). New cstate: current_term: 1 leader_uuid: "6b68c749403d4f96a817a83a1a679a8d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b68c749403d4f96a817a83a1a679a8d" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 39361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b9304cb76a2417b956f9e63e78b7734" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 35987 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "41d4efe5a7fd4b2a94dd1489ff6f3482" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 40739 } health_report { overall_health: UNKNOWN } } }
Jun 20 09:26:37 dist-test-slave-706q krb5kdc[24020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947596, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.43.190@KRBTEST.COM for oryx/127.16.43.190@KRBTEST.COM
I20260620 09:26:37.519944 24125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:37.512351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48062 (local address 127.16.43.190:36561)
0620 09:26:37.512451 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:37.512453 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:37.512545 (+ 92us) server_negotiation.cc:407] Connection header received
0620 09:26:37.512574 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:37.512575 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:37.512584 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:37.512653 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:37.513166 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:37.513661 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:37.514369 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:37.514567 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:37.514638 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:37.516979 (+ 2341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:37.516980 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:37.516980 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:26:37.516987 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:37.518524 (+ 1537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:37.518970 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:37.518971 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:37.518971 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:37.519054 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:37.519300 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:37.519301 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:37.519301 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:37.519414 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:37.519467 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:37.519752 (+ 285us) server_negotiation.cc:299] Negotiation successful
0620 09:26:37.519832 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 09:26:37.520001 24612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:37.512257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:36561 (local address 127.0.0.1:48062)
0620 09:26:37.512405 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:37.512417 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:37.512508 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:37.512700 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:37.512701 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:37.513033 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:37.513083 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:37.513085 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:37.513747 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:37.513748 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:37.514279 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:37.514281 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:37.514662 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:37.514663 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:37.514751 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:37.515354 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:26:37.515359 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:26:37.516839 (+ 1480us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:26:37.518616 (+ 1777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:37.518624 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:37.518626 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:37.518881 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:37.519159 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:37.519160 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:37.519160 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:37.519234 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:37.519492 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:26:37.519499 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:26:37.519631 (+ 132us) client_negotiation.cc:769] Sending connection context
0620 09:26:37.519703 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 09:26:37.519890 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"mutex_wait_us":28,"thread_start_us":74,"threads_started":1}
I20260620 09:26:37.522185 24064 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.16.43.190@KRBTEST.COM'} at 127.0.0.1:48062:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 09:26:37.522437 24064 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 09:26:37.524099 24337 tablet_service.cc:1467] Processing CreateTablet for tablet 1fabcee2f3ea48698c24f39a4fdba185 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3ad1d540eeb14ef7abc4c44e457496c1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 09:26:37.524081 24202 tablet_service.cc:1467] Processing CreateTablet for tablet 1fabcee2f3ea48698c24f39a4fdba185 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3ad1d540eeb14ef7abc4c44e457496c1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 09:26:37.524255 24202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fabcee2f3ea48698c24f39a4fdba185. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:26:37.524255 24337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fabcee2f3ea48698c24f39a4fdba185. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:26:37.524217 24469 tablet_service.cc:1467] Processing CreateTablet for tablet 1fabcee2f3ea48698c24f39a4fdba185 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3ad1d540eeb14ef7abc4c44e457496c1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 09:26:37.524376 24469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fabcee2f3ea48698c24f39a4fdba185. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:26:37.525655 24577 tablet_bootstrap.cc:492] T 1fabcee2f3ea48698c24f39a4fdba185 P 41d4efe5a7fd4b2a94dd1489ff6f3482: Bootstrap starting.
I20260620 09:26:37.525851 24576 tablet_bootstrap.cc:492] T 1fabcee2f3ea48698c24f39a4fdba185 P 6b68c749403d4f96a817a83a1a679a8d: Bootstrap starting.
I20260620 09:26:37.525851 24581 tablet_bootstrap.cc:492] T 1fabcee2f3ea48698c24f39a4fdba185 P 6b9304cb76a2417b956f9e63e78b7734: Bootstrap starting.
I20260620 09:26:37.526120 24577 tablet_bootstrap.cc:654] T 1fabcee2f3ea48698c24f39a4fdba185 P 41d4efe5a7fd4b2a94dd1489ff6f3482: Neither blocks nor log segments found. Creating new log.
I20260620 09:26:37.526161 24576 tablet_bootstrap.cc:654] T 1fabcee2f3ea48698c24f39a4fdba185 P 6b68c749403d4f96a817a83a1a679a8d: Neither blocks nor log segments found. Creating new log.
I20260620 09:26:37.526209 24581 tablet_bootstrap.cc:654] T 1fabcee2f3ea48698c24f39a4fdba185 P 6b9304cb76a2417b956f9e63e78b7734: Neither blocks nor log segments found. Creating new log.
I20260620 09:26:37.526868 24581 tablet_bootstrap.cc:492] T 1fabcee2f3ea48698c24f39a4fdba185 P 6b9304cb76a2417b956f9e63e78b7734: No bootstrap required, opened a new log
I20260620 09:26:37.526932 24581 ts_tablet_manager.cc:1397] T 1fabcee2f3ea48698c24f39a4fdba185 P 6b9304cb76a2417b956f9e63e78b7734: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 09:26:37.526996 24576 tablet_bootstrap.cc:492] T 1fabcee2f3ea48698c24f39a4fdba185 P 6b68c749403d4f96a817a83a1a679a8d: No bootstrap required, opened a new log
I20260620 09:26:37.527050 24576 ts_tablet_manager.cc:1397] T 1fabcee2f3ea48698c24f39a4fdba185 P 6b68c749403d4f96a817a83a1a679a8d: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260620 09:26:37.527088 24581 raft_consensus.cc:348] T 1fabcee2f3ea48698c24f39a4fdba185 P 6b9304cb76a2417b956f9e63e78b7734 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b68c749403d4f96a817a83a1a679a8d" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 39361 } } peers { permanent_uuid: "41d4efe5a7fd4b2a94dd1489ff6f3482" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 40739 } } peers { permanent_uuid: "6b9304cb76a2417b956f9e63e78b7734" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 35987 } }
I20260620 09:26:37.527170 24581 raft_consensus.cc:374] T 1fabcee2f3ea48698c24f39a4fdba185 P 6b9304cb76a2417b956f9e63e78b7734 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:26:37.527212 24581 raft_consensus.cc:729] T 1fabcee2f3ea48698c24f39a4fdba185 P 6b9304cb76a2417b956f9e63e78b7734 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b9304cb76a2417b956f9e63e78b7734, State: Initialized, Role: FOLLOWER
I20260620 09:26:37.527176 24576 raft_consensus.cc:348] T 1fabcee2f3ea48698c24f39a4fdba185 P 6b68c749403d4f96a817a83a1a679a8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b68c749403d4f96a817a83a1a679a8d" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 39361 } } peers { permanent_uuid: "41d4efe5a7fd4b2a94dd1489ff6f3482" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 40739 } } peers { permanent_uuid: "6b9304cb76a2417b956f9e63e78b7734" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 35987 } }
I20260620 09:26:37.527243 24576 raft_consensus.cc:374] T 1fabcee2f3ea48698c24f39a4fdba185 P 6b68c749403d4f96a817a83a1a679a8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:26:37.527273 24576 raft_consensus.cc:729] T 1fabcee2f3ea48698c24f39a4fdba185 P 6b68c749403d4f96a817a83a1a679a8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b68c749403d4f96a817a83a1a679a8d, State: Initialized, Role: FOLLOWER
I20260620 09:26:37.527261 24581 consensus_queue.cc:260] T 1fabcee2f3ea48698c24f39a4fdba185 P 6b9304cb76a2417b956f9e63e78b7734 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b68c749403d4f96a817a83a1a679a8d" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 39361 } } peers { permanent_uuid: "41d4efe5a7fd4b2a94dd1489ff6f3482" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 40739 } } peers { permanent_uuid: "6b9304cb76a2417b956f9e63e78b7734" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 35987 } }
I20260620 09:26:37.527324 24576 consensus_queue.cc:260] T 1fabcee2f3ea48698c24f39a4fdba185 P 6b68c749403d4f96a817a83a1a679a8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b68c749403d4f96a817a83a1a679a8d" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 39361 } } peers { permanent_uuid: "41d4efe5a7fd4b2a94dd1489ff6f3482" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 40739 } } peers { permanent_uuid: "6b9304cb76a2417b956f9e63e78b7734" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 35987 } }
I20260620 09:26:37.527354 24581 ts_tablet_manager.cc:1428] T 1fabcee2f3ea48698c24f39a4fdba185 P 6b9304cb76a2417b956f9e63e78b7734: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 09:26:37.527444 24576 ts_tablet_manager.cc:1428] T 1fabcee2f3ea48698c24f39a4fdba185 P 6b68c749403d4f96a817a83a1a679a8d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:37.527486 24584 tablet_replica.cc:439] T 1fabcee2f3ea48698c24f39a4fdba185 P 6b9304cb76a2417b956f9e63e78b7734: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b68c749403d4f96a817a83a1a679a8d" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 39361 } } peers { permanent_uuid: "41d4efe5a7fd4b2a94dd1489ff6f3482" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 40739 } } peers { permanent_uuid: "6b9304cb76a2417b956f9e63e78b7734" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 35987 } } }
I20260620 09:26:37.527622 24584 tablet_replica.cc:442] T 1fabcee2f3ea48698c24f39a4fdba185 P 6b9304cb76a2417b956f9e63e78b7734: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 09:26:37.527603 24583 tablet_replica.cc:439] T 1fabcee2f3ea48698c24f39a4fdba185 P 6b68c749403d4f96a817a83a1a679a8d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b68c749403d4f96a817a83a1a679a8d" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 39361 } } peers { permanent_uuid: "41d4efe5a7fd4b2a94dd1489ff6f3482" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 40739 } } peers { permanent_uuid: "6b9304cb76a2417b956f9e63e78b7734" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 35987 } } }
I20260620 09:26:37.527695 24583 tablet_replica.cc:442] T 1fabcee2f3ea48698c24f39a4fdba185 P 6b68c749403d4f96a817a83a1a679a8d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 09:26:37.528075 24577 tablet_bootstrap.cc:492] T 1fabcee2f3ea48698c24f39a4fdba185 P 41d4efe5a7fd4b2a94dd1489ff6f3482: No bootstrap required, opened a new log
I20260620 09:26:37.528129 24577 ts_tablet_manager.cc:1397] T 1fabcee2f3ea48698c24f39a4fdba185 P 41d4efe5a7fd4b2a94dd1489ff6f3482: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 09:26:37.528276 24577 raft_consensus.cc:348] T 1fabcee2f3ea48698c24f39a4fdba185 P 41d4efe5a7fd4b2a94dd1489ff6f3482 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b68c749403d4f96a817a83a1a679a8d" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 39361 } } peers { permanent_uuid: "41d4efe5a7fd4b2a94dd1489ff6f3482" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 40739 } } peers { permanent_uuid: "6b9304cb76a2417b956f9e63e78b7734" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 35987 } }
I20260620 09:26:37.528345 24577 raft_consensus.cc:374] T 1fabcee2f3ea48698c24f39a4fdba185 P 41d4efe5a7fd4b2a94dd1489ff6f3482 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:26:37.528366 24577 raft_consensus.cc:729] T 1fabcee2f3ea48698c24f39a4fdba185 P 41d4efe5a7fd4b2a94dd1489ff6f3482 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41d4efe5a7fd4b2a94dd1489ff6f3482, State: Initialized, Role: FOLLOWER
I20260620 09:26:37.528410 24577 consensus_queue.cc:260] T 1fabcee2f3ea48698c24f39a4fdba185 P 41d4efe5a7fd4b2a94dd1489ff6f3482 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b68c749403d4f96a817a83a1a679a8d" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 39361 } } peers { permanent_uuid: "41d4efe5a7fd4b2a94dd1489ff6f3482" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 40739 } } peers { permanent_uuid: "6b9304cb76a2417b956f9e63e78b7734" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 35987 } }
I20260620 09:26:37.528507 24577 ts_tablet_manager.cc:1428] T 1fabcee2f3ea48698c24f39a4fdba185 P 41d4efe5a7fd4b2a94dd1489ff6f3482: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:37.528565 24580 tablet_replica.cc:439] T 1fabcee2f3ea48698c24f39a4fdba185 P 41d4efe5a7fd4b2a94dd1489ff6f3482: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b68c749403d4f96a817a83a1a679a8d" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 39361 } } peers { permanent_uuid: "41d4efe5a7fd4b2a94dd1489ff6f3482" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 40739 } } peers { permanent_uuid: "6b9304cb76a2417b956f9e63e78b7734" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 35987 } } }
I20260620 09:26:37.528692 24580 tablet_replica.cc:442] T 1fabcee2f3ea48698c24f39a4fdba185 P 41d4efe5a7fd4b2a94dd1489ff6f3482: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 09:26:37.826411 24580 raft_consensus.cc:482] T 1fabcee2f3ea48698c24f39a4fdba185 P 41d4efe5a7fd4b2a94dd1489ff6f3482 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:26:37.826535 24580 raft_consensus.cc:504] T 1fabcee2f3ea48698c24f39a4fdba185 P 41d4efe5a7fd4b2a94dd1489ff6f3482 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b68c749403d4f96a817a83a1a679a8d" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 39361 } } peers { permanent_uuid: "41d4efe5a7fd4b2a94dd1489ff6f3482" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 40739 } } peers { permanent_uuid: "6b9304cb76a2417b956f9e63e78b7734" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 35987 } }
I20260620 09:26:37.826954 24580 leader_election.cc:288] T 1fabcee2f3ea48698c24f39a4fdba185 P 41d4efe5a7fd4b2a94dd1489ff6f3482 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6b68c749403d4f96a817a83a1a679a8d (127.16.43.129:39361), 6b9304cb76a2417b956f9e63e78b7734 (127.16.43.130:35987)
I20260620 09:26:37.830085 24618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:37.827211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.131:35997 (local address 127.16.43.130:35987)
0620 09:26:37.827310 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:37.827313 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:37.827410 (+ 97us) server_negotiation.cc:407] Connection header received
0620 09:26:37.827500 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:37.827501 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:37.827512 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:37.827637 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:26:37.827814 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:37.828313 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:37.829302 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:37.829689 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:37.829793 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:37.829803 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:37.829965 (+ 162us) server_negotiation.cc:299] Negotiation successful
0620 09:26:37.830006 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79,"thread_start_us":47,"threads_started":1}
I20260620 09:26:37.830085 24617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:37.827142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.130:35987 (local address 127.16.43.131:35997)
0620 09:26:37.827306 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:37.827318 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:37.827429 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:37.827646 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:37.827647 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:37.827650 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:26:37.827731 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:37.827733 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:37.828412 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:37.828413 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:37.829195 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:37.829196 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:37.829793 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:37.829794 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:37.829867 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:37.829873 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 09:26:37.829957 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 09:26:37.830002 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":66,"threads_started":1}
I20260620 09:26:37.830185 24573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:37.827253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.131:54451 (local address 127.16.43.129:39361)
0620 09:26:37.827305 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:37.827307 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:37.827411 (+ 104us) server_negotiation.cc:407] Connection header received
0620 09:26:37.827500 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:37.827501 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:37.827515 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:37.827616 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:26:37.827861 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:37.828380 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:37.829401 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:37.829809 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:37.829900 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:37.829909 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:37.830064 (+ 155us) server_negotiation.cc:299] Negotiation successful
0620 09:26:37.830108 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 09:26:37.830199 24616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:37.827175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.129:39361 (local address 127.16.43.131:54451)
0620 09:26:37.827323 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:37.827335 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:37.827430 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:37.827621 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:37.827622 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:37.827624 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:26:37.827682 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:37.827684 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:37.828486 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:37.828487 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:37.829291 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:37.829293 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:37.829899 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:37.829900 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:37.829968 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:37.829972 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 09:26:37.830045 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 09:26:37.830090 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":63,"threads_started":1}
I20260620 09:26:37.830580 24362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fabcee2f3ea48698c24f39a4fdba185" candidate_uuid: "41d4efe5a7fd4b2a94dd1489ff6f3482" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b9304cb76a2417b956f9e63e78b7734" is_pre_election: true
I20260620 09:26:37.830667 24362 raft_consensus.cc:2454] T 1fabcee2f3ea48698c24f39a4fdba185 P 6b9304cb76a2417b956f9e63e78b7734 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41d4efe5a7fd4b2a94dd1489ff6f3482 in term 0.
I20260620 09:26:37.830617 24222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fabcee2f3ea48698c24f39a4fdba185" candidate_uuid: "41d4efe5a7fd4b2a94dd1489ff6f3482" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b68c749403d4f96a817a83a1a679a8d" is_pre_election: true
I20260620 09:26:37.830801 24222 raft_consensus.cc:2454] T 1fabcee2f3ea48698c24f39a4fdba185 P 6b68c749403d4f96a817a83a1a679a8d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41d4efe5a7fd4b2a94dd1489ff6f3482 in term 0.
I20260620 09:26:37.830904 24434 leader_election.cc:302] T 1fabcee2f3ea48698c24f39a4fdba185 P 41d4efe5a7fd4b2a94dd1489ff6f3482 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41d4efe5a7fd4b2a94dd1489ff6f3482, 6b9304cb76a2417b956f9e63e78b7734; no voters:
I20260620 09:26:37.831038 24580 raft_consensus.cc:2781] T 1fabcee2f3ea48698c24f39a4fdba185 P 41d4efe5a7fd4b2a94dd1489ff6f3482 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 09:26:37.831075 24580 raft_consensus.cc:482] T 1fabcee2f3ea48698c24f39a4fdba185 P 41d4efe5a7fd4b2a94dd1489ff6f3482 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 09:26:37.831096 24580 raft_consensus.cc:3037] T 1fabcee2f3ea48698c24f39a4fdba185 P 41d4efe5a7fd4b2a94dd1489ff6f3482 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:26:37.831889 24580 raft_consensus.cc:504] T 1fabcee2f3ea48698c24f39a4fdba185 P 41d4efe5a7fd4b2a94dd1489ff6f3482 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b68c749403d4f96a817a83a1a679a8d" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 39361 } } peers { permanent_uuid: "41d4efe5a7fd4b2a94dd1489ff6f3482" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 40739 } } peers { permanent_uuid: "6b9304cb76a2417b956f9e63e78b7734" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 35987 } }
I20260620 09:26:37.832041 24580 leader_election.cc:288] T 1fabcee2f3ea48698c24f39a4fdba185 P 41d4efe5a7fd4b2a94dd1489ff6f3482 [CANDIDATE]: Term 1 election: Requested vote from peers 6b68c749403d4f96a817a83a1a679a8d (127.16.43.129:39361), 6b9304cb76a2417b956f9e63e78b7734 (127.16.43.130:35987)
I20260620 09:26:37.832223 24362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fabcee2f3ea48698c24f39a4fdba185" candidate_uuid: "41d4efe5a7fd4b2a94dd1489ff6f3482" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b9304cb76a2417b956f9e63e78b7734"
I20260620 09:26:37.832222 24222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fabcee2f3ea48698c24f39a4fdba185" candidate_uuid: "41d4efe5a7fd4b2a94dd1489ff6f3482" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b68c749403d4f96a817a83a1a679a8d"
I20260620 09:26:37.832305 24362 raft_consensus.cc:3037] T 1fabcee2f3ea48698c24f39a4fdba185 P 6b9304cb76a2417b956f9e63e78b7734 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:26:37.832304 24222 raft_consensus.cc:3037] T 1fabcee2f3ea48698c24f39a4fdba185 P 6b68c749403d4f96a817a83a1a679a8d [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:26:37.832918 24362 raft_consensus.cc:2454] T 1fabcee2f3ea48698c24f39a4fdba185 P 6b9304cb76a2417b956f9e63e78b7734 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41d4efe5a7fd4b2a94dd1489ff6f3482 in term 1.
I20260620 09:26:37.833034 24222 raft_consensus.cc:2454] T 1fabcee2f3ea48698c24f39a4fdba185 P 6b68c749403d4f96a817a83a1a679a8d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41d4efe5a7fd4b2a94dd1489ff6f3482 in term 1.
I20260620 09:26:37.833128 24434 leader_election.cc:302] T 1fabcee2f3ea48698c24f39a4fdba185 P 41d4efe5a7fd4b2a94dd1489ff6f3482 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41d4efe5a7fd4b2a94dd1489ff6f3482, 6b9304cb76a2417b956f9e63e78b7734; no voters:
I20260620 09:26:37.833220 24580 raft_consensus.cc:2781] T 1fabcee2f3ea48698c24f39a4fdba185 P 41d4efe5a7fd4b2a94dd1489ff6f3482 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:26:37.833361 24580 raft_consensus.cc:686] T 1fabcee2f3ea48698c24f39a4fdba185 P 41d4efe5a7fd4b2a94dd1489ff6f3482 [term 1 LEADER]: Becoming Leader. State: Replica: 41d4efe5a7fd4b2a94dd1489ff6f3482, State: Running, Role: LEADER
I20260620 09:26:37.833449 24580 consensus_queue.cc:237] T 1fabcee2f3ea48698c24f39a4fdba185 P 41d4efe5a7fd4b2a94dd1489ff6f3482 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b68c749403d4f96a817a83a1a679a8d" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 39361 } } peers { permanent_uuid: "41d4efe5a7fd4b2a94dd1489ff6f3482" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 40739 } } peers { permanent_uuid: "6b9304cb76a2417b956f9e63e78b7734" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 35987 } }
I20260620 09:26:37.833770 24619 tablet_replica.cc:439] T 1fabcee2f3ea48698c24f39a4fdba185 P 41d4efe5a7fd4b2a94dd1489ff6f3482: TxnStatusTablet state changed. Reason: New leader 41d4efe5a7fd4b2a94dd1489ff6f3482. Latest consensus state: current_term: 1 leader_uuid: "41d4efe5a7fd4b2a94dd1489ff6f3482" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b68c749403d4f96a817a83a1a679a8d" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 39361 } } peers { permanent_uuid: "41d4efe5a7fd4b2a94dd1489ff6f3482" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 40739 } } peers { permanent_uuid: "6b9304cb76a2417b956f9e63e78b7734" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 35987 } } }
I20260620 09:26:37.833870 24619 tablet_replica.cc:442] T 1fabcee2f3ea48698c24f39a4fdba185 P 41d4efe5a7fd4b2a94dd1489ff6f3482: This TxnStatusTablet replica's current role is: LEADER
I20260620 09:26:37.834066 24621 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 09:26:37.834213 24621 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 09:26:37.834167 24065 catalog_manager.cc:5495] T 1fabcee2f3ea48698c24f39a4fdba185 P 41d4efe5a7fd4b2a94dd1489ff6f3482 reported cstate change: term changed from 0 to 1, leader changed from <none> to 41d4efe5a7fd4b2a94dd1489ff6f3482 (127.16.43.131). New cstate: current_term: 1 leader_uuid: "41d4efe5a7fd4b2a94dd1489ff6f3482" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b68c749403d4f96a817a83a1a679a8d" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 39361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "41d4efe5a7fd4b2a94dd1489ff6f3482" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 40739 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b9304cb76a2417b956f9e63e78b7734" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 35987 } health_report { overall_health: UNKNOWN } } }
I20260620 09:26:37.885847 24612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:37.882763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.131:40739 (local address 127.0.0.1:42070)
0620 09:26:37.882868 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:37.882882 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:37.882937 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:37.883195 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:37.883196 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:37.883197 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 09:26:37.883252 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:37.883254 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:37.884164 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:37.884166 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:37.884766 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:37.884768 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:37.885303 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:37.885304 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:37.885351 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:37.885366 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 09:26:37.885686 (+ 320us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 09:26:37.885692 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 09:26:37.885722 (+ 30us) client_negotiation.cc:240] Negotiation successful
0620 09:26:37.885766 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260620 09:26:37.885851 24624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:37.882895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42070 (local address 127.16.43.131:40739)
0620 09:26:37.883021 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:37.883024 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:37.883033 (+ 9us) server_negotiation.cc:407] Connection header received
0620 09:26:37.883082 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:37.883083 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:37.883101 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:37.883190 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:26:37.883344 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:37.884076 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:37.884897 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:37.885204 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:37.885284 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:37.885486 (+ 202us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:26:37.885618 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:26:37.885694 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:37.885740 (+ 46us) server_negotiation.cc:299] Negotiation successful
0620 09:26:37.885762 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":61,"threads_started":1}
I20260620 09:26:37.887248 24362 raft_consensus.cc:1261] T 1fabcee2f3ea48698c24f39a4fdba185 P 6b9304cb76a2417b956f9e63e78b7734 [term 1 FOLLOWER]: Refusing update from remote peer 41d4efe5a7fd4b2a94dd1489ff6f3482: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:26:37.887261 24222 raft_consensus.cc:1261] T 1fabcee2f3ea48698c24f39a4fdba185 P 6b68c749403d4f96a817a83a1a679a8d [term 1 FOLLOWER]: Refusing update from remote peer 41d4efe5a7fd4b2a94dd1489ff6f3482: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:26:37.887553 24580 consensus_queue.cc:1035] T 1fabcee2f3ea48698c24f39a4fdba185 P 41d4efe5a7fd4b2a94dd1489ff6f3482 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9304cb76a2417b956f9e63e78b7734" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 35987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:26:37.887696 24619 consensus_queue.cc:1035] T 1fabcee2f3ea48698c24f39a4fdba185 P 41d4efe5a7fd4b2a94dd1489ff6f3482 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b68c749403d4f96a817a83a1a679a8d" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 39361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:26:37.888392 24583 tablet_replica.cc:439] T 1fabcee2f3ea48698c24f39a4fdba185 P 6b68c749403d4f96a817a83a1a679a8d: TxnStatusTablet state changed. Reason: New leader 41d4efe5a7fd4b2a94dd1489ff6f3482. Latest consensus state: current_term: 1 leader_uuid: "41d4efe5a7fd4b2a94dd1489ff6f3482" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b68c749403d4f96a817a83a1a679a8d" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 39361 } } peers { permanent_uuid: "41d4efe5a7fd4b2a94dd1489ff6f3482" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 40739 } } peers { permanent_uuid: "6b9304cb76a2417b956f9e63e78b7734" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 35987 } } }
I20260620 09:26:37.888497 24583 tablet_replica.cc:442] T 1fabcee2f3ea48698c24f39a4fdba185 P 6b68c749403d4f96a817a83a1a679a8d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 09:26:37.888723 24584 tablet_replica.cc:439] T 1fabcee2f3ea48698c24f39a4fdba185 P 6b9304cb76a2417b956f9e63e78b7734: TxnStatusTablet state changed. Reason: New leader 41d4efe5a7fd4b2a94dd1489ff6f3482. Latest consensus state: current_term: 1 leader_uuid: "41d4efe5a7fd4b2a94dd1489ff6f3482" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b68c749403d4f96a817a83a1a679a8d" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 39361 } } peers { permanent_uuid: "41d4efe5a7fd4b2a94dd1489ff6f3482" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 40739 } } peers { permanent_uuid: "6b9304cb76a2417b956f9e63e78b7734" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 35987 } } }
I20260620 09:26:37.888829 24584 tablet_replica.cc:442] T 1fabcee2f3ea48698c24f39a4fdba185 P 6b9304cb76a2417b956f9e63e78b7734: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 09:26:37.889387 24583 tablet_replica.cc:439] T 1fabcee2f3ea48698c24f39a4fdba185 P 6b68c749403d4f96a817a83a1a679a8d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "41d4efe5a7fd4b2a94dd1489ff6f3482" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b68c749403d4f96a817a83a1a679a8d" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 39361 } } peers { permanent_uuid: "41d4efe5a7fd4b2a94dd1489ff6f3482" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 40739 } } peers { permanent_uuid: "6b9304cb76a2417b956f9e63e78b7734" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 35987 } } }
I20260620 09:26:37.889478 24583 tablet_replica.cc:442] T 1fabcee2f3ea48698c24f39a4fdba185 P 6b68c749403d4f96a817a83a1a679a8d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 09:26:37.889662 24580 tablet_replica.cc:439] T 1fabcee2f3ea48698c24f39a4fdba185 P 41d4efe5a7fd4b2a94dd1489ff6f3482: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "41d4efe5a7fd4b2a94dd1489ff6f3482" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b68c749403d4f96a817a83a1a679a8d" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 39361 } } peers { permanent_uuid: "41d4efe5a7fd4b2a94dd1489ff6f3482" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 40739 } } peers { permanent_uuid: "6b9304cb76a2417b956f9e63e78b7734" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 35987 } } }
I20260620 09:26:37.889762 24580 tablet_replica.cc:442] T 1fabcee2f3ea48698c24f39a4fdba185 P 41d4efe5a7fd4b2a94dd1489ff6f3482: This TxnStatusTablet replica's current role is: LEADER
I20260620 09:26:37.890087 24580 tablet_replica.cc:439] T 1fabcee2f3ea48698c24f39a4fdba185 P 41d4efe5a7fd4b2a94dd1489ff6f3482: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "41d4efe5a7fd4b2a94dd1489ff6f3482" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b68c749403d4f96a817a83a1a679a8d" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 39361 } } peers { permanent_uuid: "41d4efe5a7fd4b2a94dd1489ff6f3482" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 40739 } } peers { permanent_uuid: "6b9304cb76a2417b956f9e63e78b7734" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 35987 } } }
I20260620 09:26:37.890162 24580 tablet_replica.cc:442] T 1fabcee2f3ea48698c24f39a4fdba185 P 41d4efe5a7fd4b2a94dd1489ff6f3482: This TxnStatusTablet replica's current role is: LEADER
I20260620 09:26:37.890509 24584 tablet_replica.cc:439] T 1fabcee2f3ea48698c24f39a4fdba185 P 6b9304cb76a2417b956f9e63e78b7734: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "41d4efe5a7fd4b2a94dd1489ff6f3482" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b68c749403d4f96a817a83a1a679a8d" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 39361 } } peers { permanent_uuid: "41d4efe5a7fd4b2a94dd1489ff6f3482" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 40739 } } peers { permanent_uuid: "6b9304cb76a2417b956f9e63e78b7734" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 35987 } } }
I20260620 09:26:37.890606 24584 tablet_replica.cc:442] T 1fabcee2f3ea48698c24f39a4fdba185 P 6b9304cb76a2417b956f9e63e78b7734: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 09:26:37.890594 24626 mvcc.cc:205] Tried to move back new op lower bound from 7298857360943542272 to 7298857360726392832. Current Snapshot: MvccSnapshot[applied={T|T < 7298857360943542272 or (T in {7298857360943542272})}]
I20260620 09:26:37.898669 24624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:37.895827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.129:47121 (local address 127.16.43.131:40739)
0620 09:26:37.895899 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:37.895901 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:37.895967 (+ 66us) server_negotiation.cc:407] Connection header received
0620 09:26:37.896020 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:37.896021 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:37.896033 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:37.896105 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:26:37.896257 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:37.897030 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:37.897796 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:37.898095 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:37.898173 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:37.898348 (+ 175us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:26:37.898383 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:26:37.898465 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:37.898564 (+ 99us) server_negotiation.cc:299] Negotiation successful
0620 09:26:37.898590 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 09:26:37.898676 24636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:37.895750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.131:40739 (local address 127.16.43.129:47121)
0620 09:26:37.895908 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:37.895924 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:37.895976 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:37.896134 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:37.896135 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:37.896137 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 09:26:37.896190 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:37.896192 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:37.897125 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:37.897126 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:37.897710 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:37.897712 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:37.898197 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:37.898198 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:37.898247 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:37.898262 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 09:26:37.898480 (+ 218us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 09:26:37.898483 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 09:26:37.898558 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 09:26:37.898604 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":84,"threads_started":1}
I20260620 09:26:37.900970 24502 raft_consensus.cc:1261] T be4756bd86944a7f954f515ecea00d40 P 41d4efe5a7fd4b2a94dd1489ff6f3482 [term 1 FOLLOWER]: Refusing update from remote peer 6b68c749403d4f96a817a83a1a679a8d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:26:37.900995 24362 raft_consensus.cc:1261] T be4756bd86944a7f954f515ecea00d40 P 6b9304cb76a2417b956f9e63e78b7734 [term 1 FOLLOWER]: Refusing update from remote peer 6b68c749403d4f96a817a83a1a679a8d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:26:37.901219 24598 consensus_queue.cc:1035] T be4756bd86944a7f954f515ecea00d40 P 6b68c749403d4f96a817a83a1a679a8d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9304cb76a2417b956f9e63e78b7734" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 35987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:26:37.901355 24598 consensus_queue.cc:1035] T be4756bd86944a7f954f515ecea00d40 P 6b68c749403d4f96a817a83a1a679a8d [LEADER]: Connected to new peer: Peer: permanent_uuid: "41d4efe5a7fd4b2a94dd1489ff6f3482" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 40739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:26:37.902949 24625 mvcc.cc:205] Tried to move back new op lower bound from 7298857361000558592 to 7298857359288188928. Current Snapshot: MvccSnapshot[applied={T|T < 7298857361000558592}]
I20260620 09:26:37.910957 24573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:37.907723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.131:35197 (local address 127.16.43.129:39361)
0620 09:26:37.907761 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:37.907764 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:37.907970 (+ 206us) server_negotiation.cc:407] Connection header received
0620 09:26:37.908048 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:37.908049 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:37.908059 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:37.908114 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:26:37.908512 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:37.909104 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:37.910176 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:37.910394 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:37.910478 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:37.910664 (+ 186us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:26:37.910702 (+ 38us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:26:37.910767 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:37.910846 (+ 79us) server_negotiation.cc:299] Negotiation successful
0620 09:26:37.910869 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 09:26:37.910995 24640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:37.907686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.129:39361 (local address 127.16.43.131:35197)
0620 09:26:37.907915 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:37.907928 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:37.907987 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:37.908118 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:37.908119 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:37.908121 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 09:26:37.908433 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:37.908435 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:37.909182 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:37.909183 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:37.910092 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:37.910111 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:37.910480 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:37.910481 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:37.910547 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:37.910570 (+ 23us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 09:26:37.910768 (+ 198us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 09:26:37.910771 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 09:26:37.910825 (+ 54us) client_negotiation.cc:240] Negotiation successful
0620 09:26:37.910878 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":77,"threads_started":1}
I20260620 09:26:37.919341 24064 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48052:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:26:37.919473 24064 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48052:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:26:37.920181 24064 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d0f24a0b0e524fd884b6522ce4569de5: Sending DeleteTablet for 3 replicas of tablet be4756bd86944a7f954f515ecea00d40
I20260620 09:26:37.920460 24337 tablet_service.cc:1514] Processing DeleteTablet for tablet be4756bd86944a7f954f515ecea00d40 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:26:37 UTC) from {username='oryx', principal='oryx/127.16.43.190@KRBTEST.COM'} at 127.0.0.1:47812
I20260620 09:26:37.920503 24202 tablet_service.cc:1514] Processing DeleteTablet for tablet be4756bd86944a7f954f515ecea00d40 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:26:37 UTC) from {username='oryx', principal='oryx/127.16.43.190@KRBTEST.COM'} at 127.0.0.1:51258
I20260620 09:26:37.920720 24643 tablet_replica.cc:330] T be4756bd86944a7f954f515ecea00d40 P 6b68c749403d4f96a817a83a1a679a8d: stopping tablet replica
I20260620 09:26:37.920779 24643 raft_consensus.cc:2229] T be4756bd86944a7f954f515ecea00d40 P 6b68c749403d4f96a817a83a1a679a8d [term 1 LEADER]: Raft consensus shutting down.
I20260620 09:26:37.920843 24643 raft_consensus.cc:2258] T be4756bd86944a7f954f515ecea00d40 P 6b68c749403d4f96a817a83a1a679a8d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:26:37.921105 24643 ts_tablet_manager.cc:1905] T be4756bd86944a7f954f515ecea00d40 P 6b68c749403d4f96a817a83a1a679a8d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:26:37.922756 24469 tablet_service.cc:1514] Processing DeleteTablet for tablet be4756bd86944a7f954f515ecea00d40 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:26:37 UTC) from {username='oryx', principal='oryx/127.16.43.190@KRBTEST.COM'} at 127.0.0.1:42064
I20260620 09:26:37.923019 24643 ts_tablet_manager.cc:1918] T be4756bd86944a7f954f515ecea00d40 P 6b68c749403d4f96a817a83a1a679a8d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 09:26:37.923091 24643 log.cc:1196] T be4756bd86944a7f954f515ecea00d40 P 6b68c749403d4f96a817a83a1a679a8d: Deleting WAL directory at /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781947562949448-16558-0/minicluster-data/ts-0/wal/wals/be4756bd86944a7f954f515ecea00d40
I20260620 09:26:37.923336 24643 ts_tablet_manager.cc:1939] T be4756bd86944a7f954f515ecea00d40 P 6b68c749403d4f96a817a83a1a679a8d: Deleting consensus metadata
I20260620 09:26:37.923723 24052 catalog_manager.cc:4841] TS 6b68c749403d4f96a817a83a1a679a8d (127.16.43.129:39361): tablet be4756bd86944a7f954f515ecea00d40 (table test-table [id=8d1f3ae5c68b4cf0941178e1fc50f5bd]) successfully deleted
I20260620 09:26:37.926034 24645 tablet_replica.cc:330] T be4756bd86944a7f954f515ecea00d40 P 41d4efe5a7fd4b2a94dd1489ff6f3482: stopping tablet replica
I20260620 09:26:37.926112 24645 raft_consensus.cc:2229] T be4756bd86944a7f954f515ecea00d40 P 41d4efe5a7fd4b2a94dd1489ff6f3482 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:26:37.926167 24645 raft_consensus.cc:2258] T be4756bd86944a7f954f515ecea00d40 P 41d4efe5a7fd4b2a94dd1489ff6f3482 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:26:37.926292 24642 tablet_replica.cc:330] T be4756bd86944a7f954f515ecea00d40 P 6b9304cb76a2417b956f9e63e78b7734: stopping tablet replica
I20260620 09:26:37.926358 24642 raft_consensus.cc:2229] T be4756bd86944a7f954f515ecea00d40 P 6b9304cb76a2417b956f9e63e78b7734 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:26:37.926410 24642 raft_consensus.cc:2258] T be4756bd86944a7f954f515ecea00d40 P 6b9304cb76a2417b956f9e63e78b7734 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:26:37.926683 24642 ts_tablet_manager.cc:1905] T be4756bd86944a7f954f515ecea00d40 P 6b9304cb76a2417b956f9e63e78b7734: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:26:37.926703 24645 ts_tablet_manager.cc:1905] T be4756bd86944a7f954f515ecea00d40 P 41d4efe5a7fd4b2a94dd1489ff6f3482: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:26:37.928570 24645 ts_tablet_manager.cc:1918] T be4756bd86944a7f954f515ecea00d40 P 41d4efe5a7fd4b2a94dd1489ff6f3482: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 09:26:37.928584 24642 ts_tablet_manager.cc:1918] T be4756bd86944a7f954f515ecea00d40 P 6b9304cb76a2417b956f9e63e78b7734: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 09:26:37.928640 24642 log.cc:1196] T be4756bd86944a7f954f515ecea00d40 P 6b9304cb76a2417b956f9e63e78b7734: Deleting WAL directory at /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781947562949448-16558-0/minicluster-data/ts-1/wal/wals/be4756bd86944a7f954f515ecea00d40
I20260620 09:26:37.928745 24645 log.cc:1196] T be4756bd86944a7f954f515ecea00d40 P 41d4efe5a7fd4b2a94dd1489ff6f3482: Deleting WAL directory at /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781947562949448-16558-0/minicluster-data/ts-2/wal/wals/be4756bd86944a7f954f515ecea00d40
I20260620 09:26:37.928861 24642 ts_tablet_manager.cc:1939] T be4756bd86944a7f954f515ecea00d40 P 6b9304cb76a2417b956f9e63e78b7734: Deleting consensus metadata
I20260620 09:26:37.929154 24645 ts_tablet_manager.cc:1939] T be4756bd86944a7f954f515ecea00d40 P 41d4efe5a7fd4b2a94dd1489ff6f3482: Deleting consensus metadata
I20260620 09:26:37.929342 24050 catalog_manager.cc:4841] TS 6b9304cb76a2417b956f9e63e78b7734 (127.16.43.130:35987): tablet be4756bd86944a7f954f515ecea00d40 (table test-table [id=8d1f3ae5c68b4cf0941178e1fc50f5bd]) successfully deleted
I20260620 09:26:37.929530 24049 catalog_manager.cc:4841] TS 41d4efe5a7fd4b2a94dd1489ff6f3482 (127.16.43.131:40739): tablet be4756bd86944a7f954f515ecea00d40 (table test-table [id=8d1f3ae5c68b4cf0941178e1fc50f5bd]) successfully deleted
Jun 20 09:26:37 dist-test-slave-706q krb5kdc[24020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947597, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 09:26:37 dist-test-slave-706q krb5kdc[24020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.16.43.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:26:37 dist-test-slave-706q krb5kdc[24020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.16.43.190@KRBTEST.COM, Server not found in Kerberos database
W20260620 09:26:37.945377 24125 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:26:37.938999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48074 (local address 127.16.43.190:36561)
0620 09:26:37.939076 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:37.939078 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:37.939282 (+ 204us) server_negotiation.cc:407] Connection header received
0620 09:26:37.939356 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:37.939357 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:37.939372 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:37.939448 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:37.940086 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:37.940841 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:37.941661 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:37.941951 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:37.942030 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:37.945291 (+ 3261us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48074: BlockingRecv error: recv got EOF from 127.0.0.1:48074 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Jun 20 09:26:37 dist-test-slave-706q krb5kdc[24020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947597, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.43.190@KRBTEST.COM
I20260620 09:26:37.954782 24125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:37.946352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48082 (local address 127.16.43.190:36561)
0620 09:26:37.946403 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:37.946405 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:37.946489 (+ 84us) server_negotiation.cc:407] Connection header received
0620 09:26:37.946553 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:37.946554 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:37.946565 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:37.946634 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:37.947174 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:37.948045 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:37.948773 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:37.949138 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:37.949215 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:37.951913 (+ 2698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:37.951915 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:37.951915 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:26:37.951923 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:37.953588 (+ 1665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:37.953939 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:37.953941 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:37.953941 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:37.954010 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:37.954196 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:37.954197 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:37.954197 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:37.954338 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:37.954447 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:37.954606 (+ 159us) server_negotiation.cc:299] Negotiation successful
0620 09:26:37.954687 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 09:26:37.956502 24064 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48082:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 09:26:37.956679 24064 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 09:26:37.958300 24337 tablet_service.cc:1467] Processing CreateTablet for tablet 1aefb9a9edfa4200a722562501cac910 (DEFAULT_TABLE table=test-table [id=64d392fefa884b37bcc0104fea4e48d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:26:37.958313 24202 tablet_service.cc:1467] Processing CreateTablet for tablet 1aefb9a9edfa4200a722562501cac910 (DEFAULT_TABLE table=test-table [id=64d392fefa884b37bcc0104fea4e48d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:26:37.958451 24337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1aefb9a9edfa4200a722562501cac910. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:26:37.958452 24202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1aefb9a9edfa4200a722562501cac910. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:26:37.959656 24469 tablet_service.cc:1467] Processing CreateTablet for tablet 1aefb9a9edfa4200a722562501cac910 (DEFAULT_TABLE table=test-table [id=64d392fefa884b37bcc0104fea4e48d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:26:37.959795 24469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1aefb9a9edfa4200a722562501cac910. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:26:37.960801 24577 tablet_bootstrap.cc:492] T 1aefb9a9edfa4200a722562501cac910 P 41d4efe5a7fd4b2a94dd1489ff6f3482: Bootstrap starting.
I20260620 09:26:37.960832 24576 tablet_bootstrap.cc:492] T 1aefb9a9edfa4200a722562501cac910 P 6b68c749403d4f96a817a83a1a679a8d: Bootstrap starting.
I20260620 09:26:37.961172 24576 tablet_bootstrap.cc:654] T 1aefb9a9edfa4200a722562501cac910 P 6b68c749403d4f96a817a83a1a679a8d: Neither blocks nor log segments found. Creating new log.
I20260620 09:26:37.961594 24577 tablet_bootstrap.cc:654] T 1aefb9a9edfa4200a722562501cac910 P 41d4efe5a7fd4b2a94dd1489ff6f3482: Neither blocks nor log segments found. Creating new log.
I20260620 09:26:37.961758 24581 tablet_bootstrap.cc:492] T 1aefb9a9edfa4200a722562501cac910 P 6b9304cb76a2417b956f9e63e78b7734: Bootstrap starting.
I20260620 09:26:37.962193 24581 tablet_bootstrap.cc:654] T 1aefb9a9edfa4200a722562501cac910 P 6b9304cb76a2417b956f9e63e78b7734: Neither blocks nor log segments found. Creating new log.
I20260620 09:26:37.962396 24577 tablet_bootstrap.cc:492] T 1aefb9a9edfa4200a722562501cac910 P 41d4efe5a7fd4b2a94dd1489ff6f3482: No bootstrap required, opened a new log
I20260620 09:26:37.962440 24577 ts_tablet_manager.cc:1397] T 1aefb9a9edfa4200a722562501cac910 P 41d4efe5a7fd4b2a94dd1489ff6f3482: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 09:26:37.962592 24577 raft_consensus.cc:348] T 1aefb9a9edfa4200a722562501cac910 P 41d4efe5a7fd4b2a94dd1489ff6f3482 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b68c749403d4f96a817a83a1a679a8d" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 39361 } } peers { permanent_uuid: "6b9304cb76a2417b956f9e63e78b7734" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 35987 } } peers { permanent_uuid: "41d4efe5a7fd4b2a94dd1489ff6f3482" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 40739 } }
I20260620 09:26:37.962634 24576 tablet_bootstrap.cc:492] T 1aefb9a9edfa4200a722562501cac910 P 6b68c749403d4f96a817a83a1a679a8d: No bootstrap required, opened a new log
I20260620 09:26:37.962664 24577 raft_consensus.cc:374] T 1aefb9a9edfa4200a722562501cac910 P 41d4efe5a7fd4b2a94dd1489ff6f3482 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:26:37.962677 24576 ts_tablet_manager.cc:1397] T 1aefb9a9edfa4200a722562501cac910 P 6b68c749403d4f96a817a83a1a679a8d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 09:26:37.962685 24577 raft_consensus.cc:729] T 1aefb9a9edfa4200a722562501cac910 P 41d4efe5a7fd4b2a94dd1489ff6f3482 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41d4efe5a7fd4b2a94dd1489ff6f3482, State: Initialized, Role: FOLLOWER
I20260620 09:26:37.962738 24577 consensus_queue.cc:260] T 1aefb9a9edfa4200a722562501cac910 P 41d4efe5a7fd4b2a94dd1489ff6f3482 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b68c749403d4f96a817a83a1a679a8d" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 39361 } } peers { permanent_uuid: "6b9304cb76a2417b956f9e63e78b7734" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 35987 } } peers { permanent_uuid: "41d4efe5a7fd4b2a94dd1489ff6f3482" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 40739 } }
I20260620 09:26:37.962829 24577 ts_tablet_manager.cc:1428] T 1aefb9a9edfa4200a722562501cac910 P 41d4efe5a7fd4b2a94dd1489ff6f3482: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 09:26:37.962806 24576 raft_consensus.cc:348] T 1aefb9a9edfa4200a722562501cac910 P 6b68c749403d4f96a817a83a1a679a8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b68c749403d4f96a817a83a1a679a8d" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 39361 } } peers { permanent_uuid: "6b9304cb76a2417b956f9e63e78b7734" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 35987 } } peers { permanent_uuid: "41d4efe5a7fd4b2a94dd1489ff6f3482" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 40739 } }
I20260620 09:26:37.962874 24576 raft_consensus.cc:374] T 1aefb9a9edfa4200a722562501cac910 P 6b68c749403d4f96a817a83a1a679a8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:26:37.962893 24576 raft_consensus.cc:729] T 1aefb9a9edfa4200a722562501cac910 P 6b68c749403d4f96a817a83a1a679a8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b68c749403d4f96a817a83a1a679a8d, State: Initialized, Role: FOLLOWER
I20260620 09:26:37.962965 24576 consensus_queue.cc:260] T 1aefb9a9edfa4200a722562501cac910 P 6b68c749403d4f96a817a83a1a679a8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b68c749403d4f96a817a83a1a679a8d" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 39361 } } peers { permanent_uuid: "6b9304cb76a2417b956f9e63e78b7734" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 35987 } } peers { permanent_uuid: "41d4efe5a7fd4b2a94dd1489ff6f3482" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 40739 } }
I20260620 09:26:37.963064 24576 ts_tablet_manager.cc:1428] T 1aefb9a9edfa4200a722562501cac910 P 6b68c749403d4f96a817a83a1a679a8d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:37.963502 24581 tablet_bootstrap.cc:492] T 1aefb9a9edfa4200a722562501cac910 P 6b9304cb76a2417b956f9e63e78b7734: No bootstrap required, opened a new log
I20260620 09:26:37.963603 24581 ts_tablet_manager.cc:1397] T 1aefb9a9edfa4200a722562501cac910 P 6b9304cb76a2417b956f9e63e78b7734: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 09:26:37.963763 24581 raft_consensus.cc:348] T 1aefb9a9edfa4200a722562501cac910 P 6b9304cb76a2417b956f9e63e78b7734 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b68c749403d4f96a817a83a1a679a8d" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 39361 } } peers { permanent_uuid: "6b9304cb76a2417b956f9e63e78b7734" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 35987 } } peers { permanent_uuid: "41d4efe5a7fd4b2a94dd1489ff6f3482" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 40739 } }
I20260620 09:26:37.963825 24581 raft_consensus.cc:374] T 1aefb9a9edfa4200a722562501cac910 P 6b9304cb76a2417b956f9e63e78b7734 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:26:37.963840 24581 raft_consensus.cc:729] T 1aefb9a9edfa4200a722562501cac910 P 6b9304cb76a2417b956f9e63e78b7734 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b9304cb76a2417b956f9e63e78b7734, State: Initialized, Role: FOLLOWER
I20260620 09:26:37.963892 24581 consensus_queue.cc:260] T 1aefb9a9edfa4200a722562501cac910 P 6b9304cb76a2417b956f9e63e78b7734 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b68c749403d4f96a817a83a1a679a8d" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 39361 } } peers { permanent_uuid: "6b9304cb76a2417b956f9e63e78b7734" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 35987 } } peers { permanent_uuid: "41d4efe5a7fd4b2a94dd1489ff6f3482" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 40739 } }
I20260620 09:26:37.963994 24581 ts_tablet_manager.cc:1428] T 1aefb9a9edfa4200a722562501cac910 P 6b9304cb76a2417b956f9e63e78b7734: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:38.157877 24598 raft_consensus.cc:482] T 1aefb9a9edfa4200a722562501cac910 P 6b68c749403d4f96a817a83a1a679a8d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:26:38.158032 24598 raft_consensus.cc:504] T 1aefb9a9edfa4200a722562501cac910 P 6b68c749403d4f96a817a83a1a679a8d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b68c749403d4f96a817a83a1a679a8d" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 39361 } } peers { permanent_uuid: "6b9304cb76a2417b956f9e63e78b7734" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 35987 } } peers { permanent_uuid: "41d4efe5a7fd4b2a94dd1489ff6f3482" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 40739 } }
I20260620 09:26:38.158272 24598 leader_election.cc:288] T 1aefb9a9edfa4200a722562501cac910 P 6b68c749403d4f96a817a83a1a679a8d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6b9304cb76a2417b956f9e63e78b7734 (127.16.43.130:35987), 41d4efe5a7fd4b2a94dd1489ff6f3482 (127.16.43.131:40739)
I20260620 09:26:38.158492 24362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1aefb9a9edfa4200a722562501cac910" candidate_uuid: "6b68c749403d4f96a817a83a1a679a8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b9304cb76a2417b956f9e63e78b7734" is_pre_election: true
I20260620 09:26:38.158552 24502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1aefb9a9edfa4200a722562501cac910" candidate_uuid: "6b68c749403d4f96a817a83a1a679a8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41d4efe5a7fd4b2a94dd1489ff6f3482" is_pre_election: true
I20260620 09:26:38.158627 24362 raft_consensus.cc:2454] T 1aefb9a9edfa4200a722562501cac910 P 6b9304cb76a2417b956f9e63e78b7734 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b68c749403d4f96a817a83a1a679a8d in term 0.
I20260620 09:26:38.158635 24502 raft_consensus.cc:2454] T 1aefb9a9edfa4200a722562501cac910 P 41d4efe5a7fd4b2a94dd1489ff6f3482 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b68c749403d4f96a817a83a1a679a8d in term 0.
I20260620 09:26:38.158810 24154 leader_election.cc:302] T 1aefb9a9edfa4200a722562501cac910 P 6b68c749403d4f96a817a83a1a679a8d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b68c749403d4f96a817a83a1a679a8d, 6b9304cb76a2417b956f9e63e78b7734; no voters:
I20260620 09:26:38.158919 24598 raft_consensus.cc:2781] T 1aefb9a9edfa4200a722562501cac910 P 6b68c749403d4f96a817a83a1a679a8d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 09:26:38.158965 24598 raft_consensus.cc:482] T 1aefb9a9edfa4200a722562501cac910 P 6b68c749403d4f96a817a83a1a679a8d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 09:26:38.158985 24598 raft_consensus.cc:3037] T 1aefb9a9edfa4200a722562501cac910 P 6b68c749403d4f96a817a83a1a679a8d [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:26:38.159736 24598 raft_consensus.cc:504] T 1aefb9a9edfa4200a722562501cac910 P 6b68c749403d4f96a817a83a1a679a8d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b68c749403d4f96a817a83a1a679a8d" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 39361 } } peers { permanent_uuid: "6b9304cb76a2417b956f9e63e78b7734" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 35987 } } peers { permanent_uuid: "41d4efe5a7fd4b2a94dd1489ff6f3482" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 40739 } }
I20260620 09:26:38.159905 24598 leader_election.cc:288] T 1aefb9a9edfa4200a722562501cac910 P 6b68c749403d4f96a817a83a1a679a8d [CANDIDATE]: Term 1 election: Requested vote from peers 6b9304cb76a2417b956f9e63e78b7734 (127.16.43.130:35987), 41d4efe5a7fd4b2a94dd1489ff6f3482 (127.16.43.131:40739)
I20260620 09:26:38.160059 24362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1aefb9a9edfa4200a722562501cac910" candidate_uuid: "6b68c749403d4f96a817a83a1a679a8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b9304cb76a2417b956f9e63e78b7734"
I20260620 09:26:38.160074 24502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1aefb9a9edfa4200a722562501cac910" candidate_uuid: "6b68c749403d4f96a817a83a1a679a8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41d4efe5a7fd4b2a94dd1489ff6f3482"
I20260620 09:26:38.160171 24502 raft_consensus.cc:3037] T 1aefb9a9edfa4200a722562501cac910 P 41d4efe5a7fd4b2a94dd1489ff6f3482 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:26:38.160174 24362 raft_consensus.cc:3037] T 1aefb9a9edfa4200a722562501cac910 P 6b9304cb76a2417b956f9e63e78b7734 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:26:38.160802 24502 raft_consensus.cc:2454] T 1aefb9a9edfa4200a722562501cac910 P 41d4efe5a7fd4b2a94dd1489ff6f3482 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b68c749403d4f96a817a83a1a679a8d in term 1.
I20260620 09:26:38.160820 24362 raft_consensus.cc:2454] T 1aefb9a9edfa4200a722562501cac910 P 6b9304cb76a2417b956f9e63e78b7734 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b68c749403d4f96a817a83a1a679a8d in term 1.
I20260620 09:26:38.161041 24153 leader_election.cc:302] T 1aefb9a9edfa4200a722562501cac910 P 6b68c749403d4f96a817a83a1a679a8d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41d4efe5a7fd4b2a94dd1489ff6f3482, 6b68c749403d4f96a817a83a1a679a8d; no voters:
I20260620 09:26:38.161154 24598 raft_consensus.cc:2781] T 1aefb9a9edfa4200a722562501cac910 P 6b68c749403d4f96a817a83a1a679a8d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:26:38.161242 24598 raft_consensus.cc:686] T 1aefb9a9edfa4200a722562501cac910 P 6b68c749403d4f96a817a83a1a679a8d [term 1 LEADER]: Becoming Leader. State: Replica: 6b68c749403d4f96a817a83a1a679a8d, State: Running, Role: LEADER
I20260620 09:26:38.161301 24598 consensus_queue.cc:237] T 1aefb9a9edfa4200a722562501cac910 P 6b68c749403d4f96a817a83a1a679a8d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b68c749403d4f96a817a83a1a679a8d" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 39361 } } peers { permanent_uuid: "6b9304cb76a2417b956f9e63e78b7734" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 35987 } } peers { permanent_uuid: "41d4efe5a7fd4b2a94dd1489ff6f3482" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 40739 } }
I20260620 09:26:38.161944 24064 catalog_manager.cc:5495] T 1aefb9a9edfa4200a722562501cac910 P 6b68c749403d4f96a817a83a1a679a8d reported cstate change: term changed from 0 to 1, leader changed from <none> to 6b68c749403d4f96a817a83a1a679a8d (127.16.43.129). New cstate: current_term: 1 leader_uuid: "6b68c749403d4f96a817a83a1a679a8d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b68c749403d4f96a817a83a1a679a8d" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 39361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b9304cb76a2417b956f9e63e78b7734" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 35987 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "41d4efe5a7fd4b2a94dd1489ff6f3482" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 40739 } health_report { overall_health: UNKNOWN } } }
I20260620 09:26:38.191183 24573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:38.188969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51278 (local address 127.16.43.129:39361)
0620 09:26:38.189034 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:38.189036 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:38.189045 (+ 9us) server_negotiation.cc:407] Connection header received
0620 09:26:38.189075 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:38.189076 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:38.189090 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:38.189141 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:26:38.189269 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:38.189761 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:38.190509 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:38.190722 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:38.190799 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:38.190935 (+ 136us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:26:38.190969 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:26:38.191015 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:38.191072 (+ 57us) server_negotiation.cc:299] Negotiation successful
0620 09:26:38.191096 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 09:26:38.192070 24502 raft_consensus.cc:1261] T 1aefb9a9edfa4200a722562501cac910 P 41d4efe5a7fd4b2a94dd1489ff6f3482 [term 1 FOLLOWER]: Refusing update from remote peer 6b68c749403d4f96a817a83a1a679a8d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:26:38.192116 24362 raft_consensus.cc:1261] T 1aefb9a9edfa4200a722562501cac910 P 6b9304cb76a2417b956f9e63e78b7734 [term 1 FOLLOWER]: Refusing update from remote peer 6b68c749403d4f96a817a83a1a679a8d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:26:38.192305 24583 consensus_queue.cc:1035] T 1aefb9a9edfa4200a722562501cac910 P 6b68c749403d4f96a817a83a1a679a8d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9304cb76a2417b956f9e63e78b7734" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 35987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:26:38.192397 24598 consensus_queue.cc:1035] T 1aefb9a9edfa4200a722562501cac910 P 6b68c749403d4f96a817a83a1a679a8d [LEADER]: Connected to new peer: Peer: permanent_uuid: "41d4efe5a7fd4b2a94dd1489ff6f3482" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 40739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:26:38.194633 24065 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48082:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:26:38.194744 24065 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48082:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:26:38.195581 24065 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d0f24a0b0e524fd884b6522ce4569de5: Sending DeleteTablet for 3 replicas of tablet 1aefb9a9edfa4200a722562501cac910
I20260620 09:26:38.195878 24202 tablet_service.cc:1514] Processing DeleteTablet for tablet 1aefb9a9edfa4200a722562501cac910 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:26:38 UTC) from {username='oryx', principal='oryx/127.16.43.190@KRBTEST.COM'} at 127.0.0.1:51258
I20260620 09:26:38.195880 24337 tablet_service.cc:1514] Processing DeleteTablet for tablet 1aefb9a9edfa4200a722562501cac910 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:26:38 UTC) from {username='oryx', principal='oryx/127.16.43.190@KRBTEST.COM'} at 127.0.0.1:47812
I20260620 09:26:38.195909 24469 tablet_service.cc:1514] Processing DeleteTablet for tablet 1aefb9a9edfa4200a722562501cac910 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:26:38 UTC) from {username='oryx', principal='oryx/127.16.43.190@KRBTEST.COM'} at 127.0.0.1:42064
I20260620 09:26:38.195983 24643 tablet_replica.cc:330] T 1aefb9a9edfa4200a722562501cac910 P 6b68c749403d4f96a817a83a1a679a8d: stopping tablet replica
I20260620 09:26:38.195992 24645 tablet_replica.cc:330] T 1aefb9a9edfa4200a722562501cac910 P 41d4efe5a7fd4b2a94dd1489ff6f3482: stopping tablet replica
I20260620 09:26:38.196036 24645 raft_consensus.cc:2229] T 1aefb9a9edfa4200a722562501cac910 P 41d4efe5a7fd4b2a94dd1489ff6f3482 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:26:38.196046 24643 raft_consensus.cc:2229] T 1aefb9a9edfa4200a722562501cac910 P 6b68c749403d4f96a817a83a1a679a8d [term 1 LEADER]: Raft consensus shutting down.
I20260620 09:26:38.196089 24645 raft_consensus.cc:2258] T 1aefb9a9edfa4200a722562501cac910 P 41d4efe5a7fd4b2a94dd1489ff6f3482 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:26:38.196106 24643 raft_consensus.cc:2258] T 1aefb9a9edfa4200a722562501cac910 P 6b68c749403d4f96a817a83a1a679a8d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:26:38.196182 24642 tablet_replica.cc:330] T 1aefb9a9edfa4200a722562501cac910 P 6b9304cb76a2417b956f9e63e78b7734: stopping tablet replica
I20260620 09:26:38.196256 24642 raft_consensus.cc:2229] T 1aefb9a9edfa4200a722562501cac910 P 6b9304cb76a2417b956f9e63e78b7734 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:26:38.196316 24642 raft_consensus.cc:2258] T 1aefb9a9edfa4200a722562501cac910 P 6b9304cb76a2417b956f9e63e78b7734 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:26:38.196434 24645 ts_tablet_manager.cc:1905] T 1aefb9a9edfa4200a722562501cac910 P 41d4efe5a7fd4b2a94dd1489ff6f3482: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:26:38.196498 24065 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48082:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 09:26:38.196616 24643 ts_tablet_manager.cc:1905] T 1aefb9a9edfa4200a722562501cac910 P 6b68c749403d4f96a817a83a1a679a8d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 09:26:38.196655 24065 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 09:26:38.197996 24202 tablet_service.cc:1467] Processing CreateTablet for tablet 3170a5f296354f5797e2781934d0a890 (DEFAULT_TABLE table=test-table [id=fa4083397d734b268d1c8f599bf9a25b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:26:38.198046 24469 tablet_service.cc:1467] Processing CreateTablet for tablet 3170a5f296354f5797e2781934d0a890 (DEFAULT_TABLE table=test-table [id=fa4083397d734b268d1c8f599bf9a25b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:26:38.198101 24202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3170a5f296354f5797e2781934d0a890. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:26:38.198175 24469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3170a5f296354f5797e2781934d0a890. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:26:38.198259 24643 ts_tablet_manager.cc:1918] T 1aefb9a9edfa4200a722562501cac910 P 6b68c749403d4f96a817a83a1a679a8d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:26:38.198307 24643 log.cc:1196] T 1aefb9a9edfa4200a722562501cac910 P 6b68c749403d4f96a817a83a1a679a8d: Deleting WAL directory at /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781947562949448-16558-0/minicluster-data/ts-0/wal/wals/1aefb9a9edfa4200a722562501cac910
I20260620 09:26:38.198524 24643 ts_tablet_manager.cc:1939] T 1aefb9a9edfa4200a722562501cac910 P 6b68c749403d4f96a817a83a1a679a8d: Deleting consensus metadata
I20260620 09:26:38.198961 24052 catalog_manager.cc:4841] TS 6b68c749403d4f96a817a83a1a679a8d (127.16.43.129:39361): tablet 1aefb9a9edfa4200a722562501cac910 (table test-table [id=64d392fefa884b37bcc0104fea4e48d0]) successfully deleted
I20260620 09:26:38.199359 24576 tablet_bootstrap.cc:492] T 3170a5f296354f5797e2781934d0a890 P 6b68c749403d4f96a817a83a1a679a8d: Bootstrap starting.
I20260620 09:26:38.199609 24642 ts_tablet_manager.cc:1905] T 1aefb9a9edfa4200a722562501cac910 P 6b9304cb76a2417b956f9e63e78b7734: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:26:38.199698 24576 tablet_bootstrap.cc:654] T 3170a5f296354f5797e2781934d0a890 P 6b68c749403d4f96a817a83a1a679a8d: Neither blocks nor log segments found. Creating new log.
I20260620 09:26:38.198175 24645 ts_tablet_manager.cc:1918] T 1aefb9a9edfa4200a722562501cac910 P 41d4efe5a7fd4b2a94dd1489ff6f3482: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:26:38.199856 24645 log.cc:1196] T 1aefb9a9edfa4200a722562501cac910 P 41d4efe5a7fd4b2a94dd1489ff6f3482: Deleting WAL directory at /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781947562949448-16558-0/minicluster-data/ts-2/wal/wals/1aefb9a9edfa4200a722562501cac910
I20260620 09:26:38.199966 24577 tablet_bootstrap.cc:492] T 3170a5f296354f5797e2781934d0a890 P 41d4efe5a7fd4b2a94dd1489ff6f3482: Bootstrap starting.
I20260620 09:26:38.200047 24645 ts_tablet_manager.cc:1939] T 1aefb9a9edfa4200a722562501cac910 P 41d4efe5a7fd4b2a94dd1489ff6f3482: Deleting consensus metadata
I20260620 09:26:38.200273 24577 tablet_bootstrap.cc:654] T 3170a5f296354f5797e2781934d0a890 P 41d4efe5a7fd4b2a94dd1489ff6f3482: Neither blocks nor log segments found. Creating new log.
I20260620 09:26:38.200315 24049 catalog_manager.cc:4841] TS 41d4efe5a7fd4b2a94dd1489ff6f3482 (127.16.43.131:40739): tablet 1aefb9a9edfa4200a722562501cac910 (table test-table [id=64d392fefa884b37bcc0104fea4e48d0]) successfully deleted
I20260620 09:26:38.200877 24577 tablet_bootstrap.cc:492] T 3170a5f296354f5797e2781934d0a890 P 41d4efe5a7fd4b2a94dd1489ff6f3482: No bootstrap required, opened a new log
I20260620 09:26:38.200918 24577 ts_tablet_manager.cc:1397] T 3170a5f296354f5797e2781934d0a890 P 41d4efe5a7fd4b2a94dd1489ff6f3482: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 09:26:38.201005 24577 raft_consensus.cc:348] T 3170a5f296354f5797e2781934d0a890 P 41d4efe5a7fd4b2a94dd1489ff6f3482 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b68c749403d4f96a817a83a1a679a8d" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 39361 } } peers { permanent_uuid: "6b9304cb76a2417b956f9e63e78b7734" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 35987 } } peers { permanent_uuid: "41d4efe5a7fd4b2a94dd1489ff6f3482" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 40739 } }
I20260620 09:26:38.201058 24577 raft_consensus.cc:374] T 3170a5f296354f5797e2781934d0a890 P 41d4efe5a7fd4b2a94dd1489ff6f3482 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:26:38.201071 24577 raft_consensus.cc:729] T 3170a5f296354f5797e2781934d0a890 P 41d4efe5a7fd4b2a94dd1489ff6f3482 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41d4efe5a7fd4b2a94dd1489ff6f3482, State: Initialized, Role: FOLLOWER
I20260620 09:26:38.201138 24577 consensus_queue.cc:260] T 3170a5f296354f5797e2781934d0a890 P 41d4efe5a7fd4b2a94dd1489ff6f3482 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b68c749403d4f96a817a83a1a679a8d" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 39361 } } peers { permanent_uuid: "6b9304cb76a2417b956f9e63e78b7734" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 35987 } } peers { permanent_uuid: "41d4efe5a7fd4b2a94dd1489ff6f3482" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 40739 } }
I20260620 09:26:38.201200 24642 ts_tablet_manager.cc:1918] T 1aefb9a9edfa4200a722562501cac910 P 6b9304cb76a2417b956f9e63e78b7734: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:26:38.201265 24642 log.cc:1196] T 1aefb9a9edfa4200a722562501cac910 P 6b9304cb76a2417b956f9e63e78b7734: Deleting WAL directory at /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781947562949448-16558-0/minicluster-data/ts-1/wal/wals/1aefb9a9edfa4200a722562501cac910
I20260620 09:26:38.201262 24577 ts_tablet_manager.cc:1428] T 3170a5f296354f5797e2781934d0a890 P 41d4efe5a7fd4b2a94dd1489ff6f3482: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:38.201439 24576 tablet_bootstrap.cc:492] T 3170a5f296354f5797e2781934d0a890 P 6b68c749403d4f96a817a83a1a679a8d: No bootstrap required, opened a new log
I20260620 09:26:38.201478 24576 ts_tablet_manager.cc:1397] T 3170a5f296354f5797e2781934d0a890 P 6b68c749403d4f96a817a83a1a679a8d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 09:26:38.201500 24642 ts_tablet_manager.cc:1939] T 1aefb9a9edfa4200a722562501cac910 P 6b9304cb76a2417b956f9e63e78b7734: Deleting consensus metadata
I20260620 09:26:38.201556 24576 raft_consensus.cc:348] T 3170a5f296354f5797e2781934d0a890 P 6b68c749403d4f96a817a83a1a679a8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b68c749403d4f96a817a83a1a679a8d" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 39361 } } peers { permanent_uuid: "6b9304cb76a2417b956f9e63e78b7734" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 35987 } } peers { permanent_uuid: "41d4efe5a7fd4b2a94dd1489ff6f3482" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 40739 } }
I20260620 09:26:38.201606 24576 raft_consensus.cc:374] T 3170a5f296354f5797e2781934d0a890 P 6b68c749403d4f96a817a83a1a679a8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:26:38.201619 24576 raft_consensus.cc:729] T 3170a5f296354f5797e2781934d0a890 P 6b68c749403d4f96a817a83a1a679a8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b68c749403d4f96a817a83a1a679a8d, State: Initialized, Role: FOLLOWER
I20260620 09:26:38.201683 24576 consensus_queue.cc:260] T 3170a5f296354f5797e2781934d0a890 P 6b68c749403d4f96a817a83a1a679a8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b68c749403d4f96a817a83a1a679a8d" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 39361 } } peers { permanent_uuid: "6b9304cb76a2417b956f9e63e78b7734" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 35987 } } peers { permanent_uuid: "41d4efe5a7fd4b2a94dd1489ff6f3482" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 40739 } }
I20260620 09:26:38.201785 24576 ts_tablet_manager.cc:1428] T 3170a5f296354f5797e2781934d0a890 P 6b68c749403d4f96a817a83a1a679a8d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:38.201946 24050 catalog_manager.cc:4841] TS 6b9304cb76a2417b956f9e63e78b7734 (127.16.43.130:35987): tablet 1aefb9a9edfa4200a722562501cac910 (table test-table [id=64d392fefa884b37bcc0104fea4e48d0]) successfully deleted
I20260620 09:26:38.201992 24337 tablet_service.cc:1467] Processing CreateTablet for tablet 3170a5f296354f5797e2781934d0a890 (DEFAULT_TABLE table=test-table [id=fa4083397d734b268d1c8f599bf9a25b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:26:38.202083 24337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3170a5f296354f5797e2781934d0a890. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:26:38.202921 24581 tablet_bootstrap.cc:492] T 3170a5f296354f5797e2781934d0a890 P 6b9304cb76a2417b956f9e63e78b7734: Bootstrap starting.
I20260620 09:26:38.203459 24581 tablet_bootstrap.cc:654] T 3170a5f296354f5797e2781934d0a890 P 6b9304cb76a2417b956f9e63e78b7734: Neither blocks nor log segments found. Creating new log.
I20260620 09:26:38.204239 24581 tablet_bootstrap.cc:492] T 3170a5f296354f5797e2781934d0a890 P 6b9304cb76a2417b956f9e63e78b7734: No bootstrap required, opened a new log
I20260620 09:26:38.204303 24581 ts_tablet_manager.cc:1397] T 3170a5f296354f5797e2781934d0a890 P 6b9304cb76a2417b956f9e63e78b7734: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260620 09:26:38.204439 24581 raft_consensus.cc:348] T 3170a5f296354f5797e2781934d0a890 P 6b9304cb76a2417b956f9e63e78b7734 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b68c749403d4f96a817a83a1a679a8d" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 39361 } } peers { permanent_uuid: "6b9304cb76a2417b956f9e63e78b7734" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 35987 } } peers { permanent_uuid: "41d4efe5a7fd4b2a94dd1489ff6f3482" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 40739 } }
I20260620 09:26:38.204511 24581 raft_consensus.cc:374] T 3170a5f296354f5797e2781934d0a890 P 6b9304cb76a2417b956f9e63e78b7734 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:26:38.204533 24581 raft_consensus.cc:729] T 3170a5f296354f5797e2781934d0a890 P 6b9304cb76a2417b956f9e63e78b7734 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b9304cb76a2417b956f9e63e78b7734, State: Initialized, Role: FOLLOWER
I20260620 09:26:38.204579 24581 consensus_queue.cc:260] T 3170a5f296354f5797e2781934d0a890 P 6b9304cb76a2417b956f9e63e78b7734 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b68c749403d4f96a817a83a1a679a8d" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 39361 } } peers { permanent_uuid: "6b9304cb76a2417b956f9e63e78b7734" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 35987 } } peers { permanent_uuid: "41d4efe5a7fd4b2a94dd1489ff6f3482" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 40739 } }
I20260620 09:26:38.204669 24581 ts_tablet_manager.cc:1428] T 3170a5f296354f5797e2781934d0a890 P 6b9304cb76a2417b956f9e63e78b7734: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:38.459383 24584 raft_consensus.cc:482] T 3170a5f296354f5797e2781934d0a890 P 6b9304cb76a2417b956f9e63e78b7734 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:26:38.459506 24584 raft_consensus.cc:504] T 3170a5f296354f5797e2781934d0a890 P 6b9304cb76a2417b956f9e63e78b7734 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b68c749403d4f96a817a83a1a679a8d" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 39361 } } peers { permanent_uuid: "6b9304cb76a2417b956f9e63e78b7734" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 35987 } } peers { permanent_uuid: "41d4efe5a7fd4b2a94dd1489ff6f3482" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 40739 } }
I20260620 09:26:38.459945 24584 leader_election.cc:288] T 3170a5f296354f5797e2781934d0a890 P 6b9304cb76a2417b956f9e63e78b7734 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6b68c749403d4f96a817a83a1a679a8d (127.16.43.129:39361), 41d4efe5a7fd4b2a94dd1489ff6f3482 (127.16.43.131:40739)
I20260620 09:26:38.463088 24670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:38.460229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.130:34189 (local address 127.16.43.131:40739)
0620 09:26:38.460331 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:38.460333 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:38.460373 (+ 40us) server_negotiation.cc:407] Connection header received
0620 09:26:38.460485 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:38.460486 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:38.460497 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:38.460599 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:26:38.460764 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:38.461258 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:38.462282 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:38.462690 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:38.462780 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:38.462789 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:38.462958 (+ 169us) server_negotiation.cc:299] Negotiation successful
0620 09:26:38.463002 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80,"thread_start_us":49,"threads_started":1}
I20260620 09:26:38.463088 24573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:38.460252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.130:52479 (local address 127.16.43.129:39361)
0620 09:26:38.460329 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:38.460331 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:38.460432 (+ 101us) server_negotiation.cc:407] Connection header received
0620 09:26:38.460529 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:38.460530 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:38.460540 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:38.460614 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:26:38.460778 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:38.461252 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:38.462311 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:38.462697 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:38.462784 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:38.462793 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:38.462969 (+ 176us) server_negotiation.cc:299] Negotiation successful
0620 09:26:38.463014 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 09:26:38.463140 24669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:38.460164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.131:40739 (local address 127.16.43.130:34189)
0620 09:26:38.460309 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:38.460321 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:38.460393 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:38.460601 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:38.460602 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:38.460605 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:26:38.460683 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:38.460685 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:38.461346 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:38.461347 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:38.462169 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:38.462171 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:38.462779 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:38.462780 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:38.462861 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:38.462864 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 09:26:38.462949 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 09:26:38.462996 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"mutex_wait_us":1,"thread_start_us":72,"threads_started":1}
I20260620 09:26:38.463150 24668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:38.460192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.129:39361 (local address 127.16.43.130:52479)
0620 09:26:38.460333 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:38.460352 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:38.460450 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:38.460618 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:38.460619 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:38.460622 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:26:38.460704 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:38.460706 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:38.461365 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:38.461366 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:38.462175 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:38.462177 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:38.462798 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:38.462799 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:38.462894 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:38.462897 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 09:26:38.462974 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 09:26:38.463014 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"mutex_wait_us":2,"thread_start_us":58,"threads_started":1}
I20260620 09:26:38.463466 24222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3170a5f296354f5797e2781934d0a890" candidate_uuid: "6b9304cb76a2417b956f9e63e78b7734" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b68c749403d4f96a817a83a1a679a8d" is_pre_election: true
I20260620 09:26:38.463606 24222 raft_consensus.cc:2454] T 3170a5f296354f5797e2781934d0a890 P 6b68c749403d4f96a817a83a1a679a8d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b9304cb76a2417b956f9e63e78b7734 in term 0.
I20260620 09:26:38.463574 24502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3170a5f296354f5797e2781934d0a890" candidate_uuid: "6b9304cb76a2417b956f9e63e78b7734" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41d4efe5a7fd4b2a94dd1489ff6f3482" is_pre_election: true
I20260620 09:26:38.463682 24502 raft_consensus.cc:2454] T 3170a5f296354f5797e2781934d0a890 P 41d4efe5a7fd4b2a94dd1489ff6f3482 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b9304cb76a2417b956f9e63e78b7734 in term 0.
I20260620 09:26:38.463881 24293 leader_election.cc:302] T 3170a5f296354f5797e2781934d0a890 P 6b9304cb76a2417b956f9e63e78b7734 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41d4efe5a7fd4b2a94dd1489ff6f3482, 6b9304cb76a2417b956f9e63e78b7734; no voters:
I20260620 09:26:38.464061 24584 raft_consensus.cc:2781] T 3170a5f296354f5797e2781934d0a890 P 6b9304cb76a2417b956f9e63e78b7734 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 09:26:38.464125 24584 raft_consensus.cc:482] T 3170a5f296354f5797e2781934d0a890 P 6b9304cb76a2417b956f9e63e78b7734 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 09:26:38.464175 24584 raft_consensus.cc:3037] T 3170a5f296354f5797e2781934d0a890 P 6b9304cb76a2417b956f9e63e78b7734 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:26:38.464959 24584 raft_consensus.cc:504] T 3170a5f296354f5797e2781934d0a890 P 6b9304cb76a2417b956f9e63e78b7734 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b68c749403d4f96a817a83a1a679a8d" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 39361 } } peers { permanent_uuid: "6b9304cb76a2417b956f9e63e78b7734" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 35987 } } peers { permanent_uuid: "41d4efe5a7fd4b2a94dd1489ff6f3482" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 40739 } }
I20260620 09:26:38.465140 24584 leader_election.cc:288] T 3170a5f296354f5797e2781934d0a890 P 6b9304cb76a2417b956f9e63e78b7734 [CANDIDATE]: Term 1 election: Requested vote from peers 6b68c749403d4f96a817a83a1a679a8d (127.16.43.129:39361), 41d4efe5a7fd4b2a94dd1489ff6f3482 (127.16.43.131:40739)
I20260620 09:26:38.465334 24502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3170a5f296354f5797e2781934d0a890" candidate_uuid: "6b9304cb76a2417b956f9e63e78b7734" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41d4efe5a7fd4b2a94dd1489ff6f3482"
I20260620 09:26:38.465426 24502 raft_consensus.cc:3037] T 3170a5f296354f5797e2781934d0a890 P 41d4efe5a7fd4b2a94dd1489ff6f3482 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:26:38.466030 24502 raft_consensus.cc:2454] T 3170a5f296354f5797e2781934d0a890 P 41d4efe5a7fd4b2a94dd1489ff6f3482 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b9304cb76a2417b956f9e63e78b7734 in term 1.
I20260620 09:26:38.466161 24222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3170a5f296354f5797e2781934d0a890" candidate_uuid: "6b9304cb76a2417b956f9e63e78b7734" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b68c749403d4f96a817a83a1a679a8d"
I20260620 09:26:38.466187 24293 leader_election.cc:302] T 3170a5f296354f5797e2781934d0a890 P 6b9304cb76a2417b956f9e63e78b7734 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41d4efe5a7fd4b2a94dd1489ff6f3482, 6b9304cb76a2417b956f9e63e78b7734; no voters:
I20260620 09:26:38.466265 24222 raft_consensus.cc:3037] T 3170a5f296354f5797e2781934d0a890 P 6b68c749403d4f96a817a83a1a679a8d [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:26:38.466292 24584 raft_consensus.cc:2781] T 3170a5f296354f5797e2781934d0a890 P 6b9304cb76a2417b956f9e63e78b7734 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:26:38.466446 24584 raft_consensus.cc:686] T 3170a5f296354f5797e2781934d0a890 P 6b9304cb76a2417b956f9e63e78b7734 [term 1 LEADER]: Becoming Leader. State: Replica: 6b9304cb76a2417b956f9e63e78b7734, State: Running, Role: LEADER
I20260620 09:26:38.466550 24584 consensus_queue.cc:237] T 3170a5f296354f5797e2781934d0a890 P 6b9304cb76a2417b956f9e63e78b7734 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b68c749403d4f96a817a83a1a679a8d" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 39361 } } peers { permanent_uuid: "6b9304cb76a2417b956f9e63e78b7734" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 35987 } } peers { permanent_uuid: "41d4efe5a7fd4b2a94dd1489ff6f3482" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 40739 } }
I20260620 09:26:38.467080 24222 raft_consensus.cc:2454] T 3170a5f296354f5797e2781934d0a890 P 6b68c749403d4f96a817a83a1a679a8d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b9304cb76a2417b956f9e63e78b7734 in term 1.
I20260620 09:26:38.467348 24064 catalog_manager.cc:5495] T 3170a5f296354f5797e2781934d0a890 P 6b9304cb76a2417b956f9e63e78b7734 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6b9304cb76a2417b956f9e63e78b7734 (127.16.43.130). New cstate: current_term: 1 leader_uuid: "6b9304cb76a2417b956f9e63e78b7734" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b68c749403d4f96a817a83a1a679a8d" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 39361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b9304cb76a2417b956f9e63e78b7734" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 35987 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "41d4efe5a7fd4b2a94dd1489ff6f3482" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 40739 } health_report { overall_health: UNKNOWN } } }
I20260620 09:26:38.494546 24674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:38.491845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47820 (local address 127.16.43.130:35987)
0620 09:26:38.492033 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:38.492035 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:38.492041 (+ 6us) server_negotiation.cc:407] Connection header received
0620 09:26:38.492087 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:38.492087 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:38.492108 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:38.492179 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:26:38.492311 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:38.492952 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:38.493668 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:38.493967 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:38.494030 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:38.494164 (+ 134us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:26:38.494288 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:26:38.494346 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:38.494385 (+ 39us) server_negotiation.cc:299] Negotiation successful
0620 09:26:38.494404 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":135,"threads_started":1}
I20260620 09:26:38.500041 24670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:38.497381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.130:43027 (local address 127.16.43.131:40739)
0620 09:26:38.497473 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:38.497475 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:38.497635 (+ 160us) server_negotiation.cc:407] Connection header received
0620 09:26:38.497662 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:38.497663 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:38.497779 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:38.497858 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:26:38.497978 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:38.498479 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:38.499232 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:38.499444 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:38.499597 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:38.499706 (+ 109us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:26:38.499792 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:26:38.499842 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:38.499944 (+ 102us) server_negotiation.cc:299] Negotiation successful
0620 09:26:38.499976 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 09:26:38.500059 24677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:38.497313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.131:40739 (local address 127.16.43.130:43027)
0620 09:26:38.497545 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:38.497555 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:38.497610 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:38.497859 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:38.497860 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:38.497861 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 09:26:38.497913 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:38.497914 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:38.498545 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:38.498545 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:38.499136 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:38.499138 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:38.499555 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:38.499556 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:38.499616 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:38.499629 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 09:26:38.499855 (+ 226us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 09:26:38.499857 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 09:26:38.499936 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 09:26:38.499988 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":104,"threads_started":1}
I20260620 09:26:38.502535 24502 raft_consensus.cc:1261] T 3170a5f296354f5797e2781934d0a890 P 41d4efe5a7fd4b2a94dd1489ff6f3482 [term 1 FOLLOWER]: Refusing update from remote peer 6b9304cb76a2417b956f9e63e78b7734: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:26:38.502552 24222 raft_consensus.cc:1261] T 3170a5f296354f5797e2781934d0a890 P 6b68c749403d4f96a817a83a1a679a8d [term 1 FOLLOWER]: Refusing update from remote peer 6b9304cb76a2417b956f9e63e78b7734: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:26:38.502830 24584 consensus_queue.cc:1035] T 3170a5f296354f5797e2781934d0a890 P 6b9304cb76a2417b956f9e63e78b7734 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41d4efe5a7fd4b2a94dd1489ff6f3482" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 40739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:26:38.503003 24671 consensus_queue.cc:1035] T 3170a5f296354f5797e2781934d0a890 P 6b9304cb76a2417b956f9e63e78b7734 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b68c749403d4f96a817a83a1a679a8d" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 39361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:26:38.510003 24674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:38.507360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.131:33245 (local address 127.16.43.130:35987)
0620 09:26:38.507400 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:38.507403 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:38.507498 (+ 95us) server_negotiation.cc:407] Connection header received
0620 09:26:38.507599 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:38.507600 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:38.507611 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:38.507671 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:26:38.507801 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:38.508277 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:38.509218 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:38.509541 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:38.509608 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:38.509737 (+ 129us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:26:38.509772 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:26:38.509820 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:38.509927 (+ 107us) server_negotiation.cc:299] Negotiation successful
0620 09:26:38.509946 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 09:26:38.510032 24684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:38.507298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.130:35987 (local address 127.16.43.131:33245)
0620 09:26:38.507417 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:38.507428 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:38.507513 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:38.507666 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:38.507667 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:38.507668 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 09:26:38.507753 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:38.507754 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:38.508341 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:38.508342 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:38.509113 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:38.509115 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:38.509613 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:38.509614 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:38.509665 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:38.509669 (+ 4us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 09:26:38.509856 (+ 187us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 09:26:38.509858 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 09:26:38.509906 (+ 48us) client_negotiation.cc:240] Negotiation successful
0620 09:26:38.509954 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":70,"threads_started":1}
I20260620 09:26:38.520052 24065 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48082:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:26:38.520175 24065 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48082:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:26:38.520860 24065 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d0f24a0b0e524fd884b6522ce4569de5: Sending DeleteTablet for 3 replicas of tablet 3170a5f296354f5797e2781934d0a890
I20260620 09:26:38.521152 24337 tablet_service.cc:1514] Processing DeleteTablet for tablet 3170a5f296354f5797e2781934d0a890 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:26:38 UTC) from {username='oryx', principal='oryx/127.16.43.190@KRBTEST.COM'} at 127.0.0.1:47812
I20260620 09:26:38.521191 24469 tablet_service.cc:1514] Processing DeleteTablet for tablet 3170a5f296354f5797e2781934d0a890 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:26:38 UTC) from {username='oryx', principal='oryx/127.16.43.190@KRBTEST.COM'} at 127.0.0.1:42064
I20260620 09:26:38.521245 24642 tablet_replica.cc:330] T 3170a5f296354f5797e2781934d0a890 P 6b9304cb76a2417b956f9e63e78b7734: stopping tablet replica
I20260620 09:26:38.521270 24645 tablet_replica.cc:330] T 3170a5f296354f5797e2781934d0a890 P 41d4efe5a7fd4b2a94dd1489ff6f3482: stopping tablet replica
I20260620 09:26:38.521235 24202 tablet_service.cc:1514] Processing DeleteTablet for tablet 3170a5f296354f5797e2781934d0a890 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:26:38 UTC) from {username='oryx', principal='oryx/127.16.43.190@KRBTEST.COM'} at 127.0.0.1:51258
I20260620 09:26:38.521294 24642 raft_consensus.cc:2229] T 3170a5f296354f5797e2781934d0a890 P 6b9304cb76a2417b956f9e63e78b7734 [term 1 LEADER]: Raft consensus shutting down.
I20260620 09:26:38.521327 24645 raft_consensus.cc:2229] T 3170a5f296354f5797e2781934d0a890 P 41d4efe5a7fd4b2a94dd1489ff6f3482 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:26:38.521341 24643 tablet_replica.cc:330] T 3170a5f296354f5797e2781934d0a890 P 6b68c749403d4f96a817a83a1a679a8d: stopping tablet replica
I20260620 09:26:38.521361 24645 raft_consensus.cc:2258] T 3170a5f296354f5797e2781934d0a890 P 41d4efe5a7fd4b2a94dd1489ff6f3482 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:26:38.521389 24643 raft_consensus.cc:2229] T 3170a5f296354f5797e2781934d0a890 P 6b68c749403d4f96a817a83a1a679a8d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:26:38.521386 24642 raft_consensus.cc:2258] T 3170a5f296354f5797e2781934d0a890 P 6b9304cb76a2417b956f9e63e78b7734 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:26:38.521423 24643 raft_consensus.cc:2258] T 3170a5f296354f5797e2781934d0a890 P 6b68c749403d4f96a817a83a1a679a8d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:26:38.521798 24645 ts_tablet_manager.cc:1905] T 3170a5f296354f5797e2781934d0a890 P 41d4efe5a7fd4b2a94dd1489ff6f3482: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:26:38.521876 24642 ts_tablet_manager.cc:1905] T 3170a5f296354f5797e2781934d0a890 P 6b9304cb76a2417b956f9e63e78b7734: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:26:38.523176 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 24137
I20260620 09:26:38.524358 24642 ts_tablet_manager.cc:1918] T 3170a5f296354f5797e2781934d0a890 P 6b9304cb76a2417b956f9e63e78b7734: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 09:26:38.524436 24642 log.cc:1196] T 3170a5f296354f5797e2781934d0a890 P 6b9304cb76a2417b956f9e63e78b7734: Deleting WAL directory at /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781947562949448-16558-0/minicluster-data/ts-1/wal/wals/3170a5f296354f5797e2781934d0a890
I20260620 09:26:38.524650 24642 ts_tablet_manager.cc:1939] T 3170a5f296354f5797e2781934d0a890 P 6b9304cb76a2417b956f9e63e78b7734: Deleting consensus metadata
I20260620 09:26:38.524976 24050 catalog_manager.cc:4841] TS 6b9304cb76a2417b956f9e63e78b7734 (127.16.43.130:35987): tablet 3170a5f296354f5797e2781934d0a890 (table test-table [id=fa4083397d734b268d1c8f599bf9a25b]) successfully deleted
I20260620 09:26:38.527695 24645 ts_tablet_manager.cc:1918] T 3170a5f296354f5797e2781934d0a890 P 41d4efe5a7fd4b2a94dd1489ff6f3482: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 09:26:38.527765 24645 log.cc:1196] T 3170a5f296354f5797e2781934d0a890 P 41d4efe5a7fd4b2a94dd1489ff6f3482: Deleting WAL directory at /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781947562949448-16558-0/minicluster-data/ts-2/wal/wals/3170a5f296354f5797e2781934d0a890
I20260620 09:26:38.527989 24645 ts_tablet_manager.cc:1939] T 3170a5f296354f5797e2781934d0a890 P 41d4efe5a7fd4b2a94dd1489ff6f3482: Deleting consensus metadata
I20260620 09:26:38.528349 24049 catalog_manager.cc:4841] TS 41d4efe5a7fd4b2a94dd1489ff6f3482 (127.16.43.131:40739): tablet 3170a5f296354f5797e2781934d0a890 (table test-table [id=fa4083397d734b268d1c8f599bf9a25b]) successfully deleted
W20260620 09:26:38.530882 24052 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.43.129:39361 (error 108)
I20260620 09:26:38.531188 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 24277
I20260620 09:26:38.531929 24688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:38.531274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.129:39361 (local address 127.0.0.1:51288)
0620 09:26:38.531786 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:38.531833 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.129:39361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":42,"threads_started":1}
W20260620 09:26:38.532021 24052 catalog_manager.cc:4584] TS 6b68c749403d4f96a817a83a1a679a8d (127.16.43.129:39361): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3170a5f296354f5797e2781934d0a890: Network error: Client connection negotiation failed: client connection to 127.16.43.129:39361: connect: Connection refused (error 111)
I20260620 09:26:38.536814 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 24417
I20260620 09:26:38.543023 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 24033
2026-06-20T09:26:38Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3338 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781947562949448-16558-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:26:38 dist-test-slave-706q krb5kdc[24691](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:26:38 dist-test-slave-706q krb5kdc[24691](info): set up 2 sockets
Jun 20 09:26:38 dist-test-slave-706q krb5kdc[24691](info): commencing operation
krb5kdc: starting...
W20260620 09:26:39.581209 16558 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781947562949448-16558-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781947562949448-16558-0/krb5kdc/test-user.keytab.
Jun 20 09:26:39 dist-test-slave-706q krb5kdc[24691](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947599, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:26:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:26:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.43.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.43.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.43.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.190.keytab.
Entry for principal kudu/127.16.43.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.190.keytab.
I20260620 09:26:39.704571 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781947562949448-16558-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781947562949448-16558-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781947562949448-16558-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.43.190:39839
--webserver_interface=127.16.43.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.190.keytab
--principal=kudu/127.16.43.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.43.148:40017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.43.190:39839
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781947562949448-16558-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781947562949448-16558-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781947562949448-16558-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:26:39.787325 24704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:26:39.787551 24704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:26:39.787575 24704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:26:39.788750 24704 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:26:39.788793 24704 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:26:39.788808 24704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:26:39.788820 24704 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:26:39.788833 24704 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:26:39.790453 24704 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:40017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781947562949448-16558-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781947562949448-16558-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.43.190:39839
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.190.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.43.190:39839
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.43.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24704
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781947562949448-16558-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:26:39.790735 24704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:26:39.790997 24704 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:26:39.796041 24710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:26:39.796041 24712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:26:39.796070 24704 server_base.cc:1034] running on GCE node
W20260620 09:26:39.796080 24709 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:26:39.796501 24704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:26:39.796840 24704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:26:39.798015 24704 hybrid_clock.cc:648] HybridClock initialized: now 1781947599798007 us; error 51 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-20T09:26:39Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1258 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:26:39 dist-test-slave-706q krb5kdc[24717](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:26:39 dist-test-slave-706q krb5kdc[24717](info): set up 2 sockets
Jun 20 09:26:39 dist-test-slave-706q krb5kdc[24717](info): commencing operation
krb5kdc: starting...
W20260620 09:26:40.853102 16558 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/krb5kdc/test-user.keytab.
Jun 20 09:26:40 dist-test-slave-706q krb5kdc[24717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947600, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:26:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:26:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.43.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.43.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.43.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.190.keytab.
Entry for principal kudu/127.16.43.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.190.keytab.
I20260620 09:26:40.978005 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.43.190:36415
--webserver_interface=127.16.43.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.190.keytab
--principal=kudu/127.16.43.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.43.148:37741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.43.190:36415
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:26:41.059170 24730 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:26:41.059379 24730 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:26:41.059402 24730 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:26:41.060604 24730 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:26:41.060652 24730 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:26:41.060667 24730 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:26:41.060680 24730 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:26:41.060693 24730 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:26:41.062206 24730 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:37741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.43.190:36415
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.190.keytab
--principal=kudu/127.16.43.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.43.190:36415
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.43.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24730
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:26:41.062467 24730 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:26:41.062704 24730 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:26:41.067636 24738 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:26:41.067665 24735 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:26:41.067665 24736 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:26:41.067708 24730 server_base.cc:1034] running on GCE node
I20260620 09:26:41.068137 24730 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:26:41.068425 24730 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:26:41.069582 24730 hybrid_clock.cc:648] HybridClock initialized: now 1781947601069586 us; error 46 us; skew 500 ppm
Jun 20 09:26:41 dist-test-slave-706q krb5kdc[24717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947601, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:26:41.075323 24730 init.cc:377] Logged in from keytab as kudu/127.16.43.190@KRBTEST.COM (short username kudu)
I20260620 09:26:41.075845 24730 webserver.cc:460] Webserver started at http://127.16.43.190:38249/ using document root <none> and password file <none>
I20260620 09:26:41.076046 24730 fs_manager.cc:359] Metadata directory not provided
I20260620 09:26:41.076089 24730 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:26:41.076177 24730 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:26:41.077219 24730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/minicluster-data/master-0/data/instance:
uuid: "de251ca09785457c9c81526d5bcde08a"
format_stamp: "Formatted at 2026-06-20 09:26:41 on dist-test-slave-706q"
server_key: "2a9c613956e2bc4c1849c09a15861e35"
server_key_iv: "c5ba19957e07378564059d09eaa5671d"
server_key_version: "encryptionkey@0"
I20260620 09:26:41.077525 24730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/minicluster-data/master-0/wal/instance:
uuid: "de251ca09785457c9c81526d5bcde08a"
format_stamp: "Formatted at 2026-06-20 09:26:41 on dist-test-slave-706q"
server_key: "2a9c613956e2bc4c1849c09a15861e35"
server_key_iv: "c5ba19957e07378564059d09eaa5671d"
server_key_version: "encryptionkey@0"
I20260620 09:26:41.078809 24730 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 09:26:41.079615 24745 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:41.079847 24730 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:26:41.079945 24730 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/minicluster-data/master-0/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/minicluster-data/master-0/wal
uuid: "de251ca09785457c9c81526d5bcde08a"
format_stamp: "Formatted at 2026-06-20 09:26:41 on dist-test-slave-706q"
server_key: "2a9c613956e2bc4c1849c09a15861e35"
server_key_iv: "c5ba19957e07378564059d09eaa5671d"
server_key_version: "encryptionkey@0"
I20260620 09:26:41.080005 24730 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:26:41.091269 24730 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:26:41.093448 24730 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:26:41.093564 24730 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:26:41.097739 24730 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.190:36415
I20260620 09:26:41.097815 24797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.190:36415 every 8 connection(s)
I20260620 09:26:41.098160 24730 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
I20260620 09:26:41.099308 24798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:26:41.101811 24798 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de251ca09785457c9c81526d5bcde08a: Bootstrap starting.
I20260620 09:26:41.102432 24798 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P de251ca09785457c9c81526d5bcde08a: Neither blocks nor log segments found. Creating new log.
I20260620 09:26:41.102766 24798 log.cc:824] T 00000000000000000000000000000000 P de251ca09785457c9c81526d5bcde08a: Log is configured to *not* fsync() on all Append() calls
I20260620 09:26:41.103313 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 24730
I20260620 09:26:41.103444 16558 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/minicluster-data/master-0/wal/instance
I20260620 09:26:41.103642 24798 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de251ca09785457c9c81526d5bcde08a: No bootstrap required, opened a new log
I20260620 09:26:41.103729 16558 external_mini_cluster.cc:1428] Setting key 00b64b137cc896663263eab03fac341f
I20260620 09:26:41.105494 24798 raft_consensus.cc:348] T 00000000000000000000000000000000 P de251ca09785457c9c81526d5bcde08a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de251ca09785457c9c81526d5bcde08a" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 36415 } }
I20260620 09:26:41.105661 24798 raft_consensus.cc:374] T 00000000000000000000000000000000 P de251ca09785457c9c81526d5bcde08a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:26:41.105723 24798 raft_consensus.cc:729] T 00000000000000000000000000000000 P de251ca09785457c9c81526d5bcde08a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de251ca09785457c9c81526d5bcde08a, State: Initialized, Role: FOLLOWER
I20260620 09:26:41.105865 24798 consensus_queue.cc:260] T 00000000000000000000000000000000 P de251ca09785457c9c81526d5bcde08a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de251ca09785457c9c81526d5bcde08a" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 36415 } }
I20260620 09:26:41.105969 24798 raft_consensus.cc:388] T 00000000000000000000000000000000 P de251ca09785457c9c81526d5bcde08a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:26:41.106014 24798 raft_consensus.cc:482] T 00000000000000000000000000000000 P de251ca09785457c9c81526d5bcde08a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:26:41.106063 24798 raft_consensus.cc:3037] T 00000000000000000000000000000000 P de251ca09785457c9c81526d5bcde08a [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:26:41.106935 24798 raft_consensus.cc:504] T 00000000000000000000000000000000 P de251ca09785457c9c81526d5bcde08a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de251ca09785457c9c81526d5bcde08a" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 36415 } }
I20260620 09:26:41.107057 24798 leader_election.cc:302] T 00000000000000000000000000000000 P de251ca09785457c9c81526d5bcde08a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: de251ca09785457c9c81526d5bcde08a; no voters:
I20260620 09:26:41.107240 24798 leader_election.cc:288] T 00000000000000000000000000000000 P de251ca09785457c9c81526d5bcde08a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:26:41.107285 24803 raft_consensus.cc:2781] T 00000000000000000000000000000000 P de251ca09785457c9c81526d5bcde08a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:26:41.107450 24803 raft_consensus.cc:686] T 00000000000000000000000000000000 P de251ca09785457c9c81526d5bcde08a [term 1 LEADER]: Becoming Leader. State: Replica: de251ca09785457c9c81526d5bcde08a, State: Running, Role: LEADER
I20260620 09:26:41.107508 24798 sys_catalog.cc:564] T 00000000000000000000000000000000 P de251ca09785457c9c81526d5bcde08a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:26:41.107632 24803 consensus_queue.cc:237] T 00000000000000000000000000000000 P de251ca09785457c9c81526d5bcde08a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de251ca09785457c9c81526d5bcde08a" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 36415 } }
I20260620 09:26:41.108042 24803 sys_catalog.cc:455] T 00000000000000000000000000000000 P de251ca09785457c9c81526d5bcde08a [sys.catalog]: SysCatalogTable state changed. Reason: New leader de251ca09785457c9c81526d5bcde08a. Latest consensus state: current_term: 1 leader_uuid: "de251ca09785457c9c81526d5bcde08a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de251ca09785457c9c81526d5bcde08a" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 36415 } } }
I20260620 09:26:41.108057 24804 sys_catalog.cc:455] T 00000000000000000000000000000000 P de251ca09785457c9c81526d5bcde08a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "de251ca09785457c9c81526d5bcde08a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de251ca09785457c9c81526d5bcde08a" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 36415 } } }
I20260620 09:26:41.108250 24804 sys_catalog.cc:458] T 00000000000000000000000000000000 P de251ca09785457c9c81526d5bcde08a [sys.catalog]: This master's current role is: LEADER
I20260620 09:26:41.108426 24803 sys_catalog.cc:458] T 00000000000000000000000000000000 P de251ca09785457c9c81526d5bcde08a [sys.catalog]: This master's current role is: LEADER
I20260620 09:26:41.108583 24812 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:26:41.109292 24812 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:26:41.111176 24812 catalog_manager.cc:1347] Generated new cluster ID: 98a0d94d2db343ffa2b8c2ab8defabc4
I20260620 09:26:41.111231 24812 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 09:26:41 dist-test-slave-706q krb5kdc[24717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947600, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.43.190@KRBTEST.COM
I20260620 09:26:41.116837 24802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:41.104394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54342 (local address 127.16.43.190:36415)
0620 09:26:41.104566 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:41.104578 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:41.104664 (+ 86us) server_negotiation.cc:407] Connection header received
0620 09:26:41.105484 (+ 820us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:41.105493 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:41.105549 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:41.105714 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:41.106507 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:41.107700 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:41.108846 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:41.109495 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:41.109709 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:41.112807 (+ 3098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:41.112812 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:41.112812 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:26:41.112825 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:41.114924 (+ 2099us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:41.115365 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:41.115369 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:41.115369 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:41.115497 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:41.115755 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:41.115756 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:41.115756 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:41.116063 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:41.116252 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:41.116382 (+ 130us) server_negotiation.cc:299] Negotiation successful
0620 09:26:41.116576 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":76,"threads_started":1}
I20260620 09:26:41.116945 24812 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:26:41.117817 24812 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:26:41.121230 24812 catalog_manager.cc:5868] T 00000000000000000000000000000000 P de251ca09785457c9c81526d5bcde08a: Generated new TSK 0
I20260620 09:26:41.121414 24812 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.43.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.43.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.43.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.129.keytab.
Entry for principal kudu/127.16.43.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.129.keytab.
I20260620 09:26:41.154954 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.43.129:0
--local_ip_for_outbound_sockets=127.16.43.129
--webserver_interface=127.16.43.129
--webserver_port=0
--tserver_master_addrs=127.16.43.190:36415
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.129.keytab
--principal=kudu/127.16.43.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.43.148:37741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:26:41.253614 24824 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:26:41.253841 24824 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:26:41.253867 24824 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:26:41.255102 24824 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:26:41.255146 24824 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:26:41.255172 24824 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.43.129
I20260620 09:26:41.256711 24824 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:37741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.129.keytab
--principal=kudu/127.16.43.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.43.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.43.129
--webserver_port=0
--tserver_master_addrs=127.16.43.190:36415
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24824
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.43.129
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:26:41.256980 24824 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:26:41.257251 24824 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:26:41.262603 24832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:26:41.262617 24829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:26:41.262607 24830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:26:41.262967 24824 server_base.cc:1034] running on GCE node
I20260620 09:26:41.263305 24824 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:26:41.263571 24824 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:26:41.264750 24824 hybrid_clock.cc:648] HybridClock initialized: now 1781947601264740 us; error 40 us; skew 500 ppm
Jun 20 09:26:41 dist-test-slave-706q krb5kdc[24717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947601, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:26:41.267411 24824 init.cc:377] Logged in from keytab as kudu/127.16.43.129@KRBTEST.COM (short username kudu)
I20260620 09:26:41.267935 24824 webserver.cc:460] Webserver started at http://127.16.43.129:34293/ using document root <none> and password file <none>
I20260620 09:26:41.268193 24824 fs_manager.cc:359] Metadata directory not provided
I20260620 09:26:41.268262 24824 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:26:41.268412 24824 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:26:41.269529 24824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/minicluster-data/ts-0/data/instance:
uuid: "f37759d2ae5a4d1a85dfa10108c5092c"
format_stamp: "Formatted at 2026-06-20 09:26:41 on dist-test-slave-706q"
server_key: "bf3760c9ddc85f105f27d191a5034f61"
server_key_iv: "0d51c6552c24da6684c3ae06c9f0a233"
server_key_version: "encryptionkey@0"
I20260620 09:26:41.269897 24824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/minicluster-data/ts-0/wal/instance:
uuid: "f37759d2ae5a4d1a85dfa10108c5092c"
format_stamp: "Formatted at 2026-06-20 09:26:41 on dist-test-slave-706q"
server_key: "bf3760c9ddc85f105f27d191a5034f61"
server_key_iv: "0d51c6552c24da6684c3ae06c9f0a233"
server_key_version: "encryptionkey@0"
I20260620 09:26:41.271288 24824 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 09:26:41.272126 24839 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:41.272337 24824 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 09:26:41.272439 24824 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/minicluster-data/ts-0/wal
uuid: "f37759d2ae5a4d1a85dfa10108c5092c"
format_stamp: "Formatted at 2026-06-20 09:26:41 on dist-test-slave-706q"
server_key: "bf3760c9ddc85f105f27d191a5034f61"
server_key_iv: "0d51c6552c24da6684c3ae06c9f0a233"
server_key_version: "encryptionkey@0"
I20260620 09:26:41.272519 24824 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:26:41.291373 24824 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:26:41.294214 24824 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:26:41.294435 24824 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:26:41.294739 24824 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:26:41.295169 24824 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:26:41.295225 24824 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:41.295265 24824 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:26:41.295308 24824 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:41.301326 24824 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.129:44231
I20260620 09:26:41.301386 24952 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.129:44231 every 8 connection(s)
I20260620 09:26:41.301749 24824 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/minicluster-data/ts-0/data/info.pb
Jun 20 09:26:41 dist-test-slave-706q krb5kdc[24717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947601, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.129@KRBTEST.COM for kudu/127.16.43.190@KRBTEST.COM
I20260620 09:26:41.311507 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 24824
I20260620 09:26:41.311656 16558 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/minicluster-data/ts-0/wal/instance
I20260620 09:26:41.311890 16558 external_mini_cluster.cc:1428] Setting key 951d4ae3f7e2753a750dfbbb8f29654b
I20260620 09:26:41.312438 24802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:41.302698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.129:53661 (local address 127.16.43.190:36415)
0620 09:26:41.302778 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:41.302780 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:41.303108 (+ 328us) server_negotiation.cc:407] Connection header received
0620 09:26:41.303641 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:41.303642 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:41.303653 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:41.303714 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:41.304764 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:41.305311 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:41.306363 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:41.306575 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:41.306628 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:41.309283 (+ 2655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:41.309284 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:41.309285 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:26:41.309300 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:41.310757 (+ 1457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:41.311183 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:41.311184 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:41.311184 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:41.311240 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:41.311620 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:41.311621 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:41.311622 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:41.311788 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:41.311911 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:41.312273 (+ 362us) server_negotiation.cc:299] Negotiation successful
0620 09:26:41.312371 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 09:26:41.312692 24955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:41.302662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:36415 (local address 127.16.43.129:53661)
0620 09:26:41.302868 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:41.302941 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:41.303430 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:41.303799 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:41.303804 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:41.304452 (+ 648us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:41.304661 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:41.304666 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:41.305409 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:41.305410 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:41.306286 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:41.306289 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:41.306649 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:41.306650 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:41.306751 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:41.307438 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:26:41.307446 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:26:41.309143 (+ 1697us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:26:41.310877 (+ 1734us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:41.310883 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:41.310885 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:41.311115 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:41.311331 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:41.311332 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:41.311332 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:41.311485 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:41.311945 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:26:41.311953 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:26:41.312137 (+ 184us) client_negotiation.cc:769] Sending connection context
0620 09:26:41.312233 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 09:26:41.312424 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":105,"threads_started":1}
I20260620 09:26:41.313235 24953 heartbeater.cc:347] Connected to a master server at 127.16.43.190:36415
I20260620 09:26:41.313367 24953 heartbeater.cc:464] Registering TS with master...
I20260620 09:26:41.313566 24953 heartbeater.cc:511] Master 127.16.43.190:36415 requested a full tablet report, sending...
I20260620 09:26:41.314116 24762 ts_manager.cc:194] Registered new tserver with Master: f37759d2ae5a4d1a85dfa10108c5092c (127.16.43.129:44231)
I20260620 09:26:41.314802 24762 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.43.129@KRBTEST.COM'} at 127.16.43.129:53661
WARNING: no policy specified for kudu/127.16.43.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.43.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.43.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.130.keytab.
Entry for principal kudu/127.16.43.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.130.keytab.
I20260620 09:26:41.338331 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.43.130:0
--local_ip_for_outbound_sockets=127.16.43.130
--webserver_interface=127.16.43.130
--webserver_port=0
--tserver_master_addrs=127.16.43.190:36415
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.130.keytab
--principal=kudu/127.16.43.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.43.148:37741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:26:41.421027 24958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:26:41.421268 24958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:26:41.421308 24958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:26:41.422784 24958 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:26:41.422871 24958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:26:41.422936 24958 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.43.130
I20260620 09:26:41.424731 24958 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:37741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.130.keytab
--principal=kudu/127.16.43.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.43.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.43.130
--webserver_port=0
--tserver_master_addrs=127.16.43.190:36415
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24958
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.43.130
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:26:41.425034 24958 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:26:41.425267 24958 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:26:41.430384 24966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:26:41.430390 24963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:26:41.430382 24964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:26:41.430640 24958 server_base.cc:1034] running on GCE node
I20260620 09:26:41.430950 24958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:26:41.431214 24958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:26:41.432376 24958 hybrid_clock.cc:648] HybridClock initialized: now 1781947601432372 us; error 38 us; skew 500 ppm
Jun 20 09:26:41 dist-test-slave-706q krb5kdc[24717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947601, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:26:41.435105 24958 init.cc:377] Logged in from keytab as kudu/127.16.43.130@KRBTEST.COM (short username kudu)
I20260620 09:26:41.435599 24958 webserver.cc:460] Webserver started at http://127.16.43.130:42213/ using document root <none> and password file <none>
I20260620 09:26:41.435846 24958 fs_manager.cc:359] Metadata directory not provided
I20260620 09:26:41.435921 24958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:26:41.436069 24958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:26:41.437135 24958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/minicluster-data/ts-1/data/instance:
uuid: "a740d9f90077438fa30c153b8967c788"
format_stamp: "Formatted at 2026-06-20 09:26:41 on dist-test-slave-706q"
server_key: "d521f39970ce9f1080450264ad0b9d32"
server_key_iv: "d7f77a60fa1a445d8e411acc2c28868d"
server_key_version: "encryptionkey@0"
I20260620 09:26:41.437494 24958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/minicluster-data/ts-1/wal/instance:
uuid: "a740d9f90077438fa30c153b8967c788"
format_stamp: "Formatted at 2026-06-20 09:26:41 on dist-test-slave-706q"
server_key: "d521f39970ce9f1080450264ad0b9d32"
server_key_iv: "d7f77a60fa1a445d8e411acc2c28868d"
server_key_version: "encryptionkey@0"
I20260620 09:26:41.438835 24958 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 09:26:41.439639 24973 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:41.439812 24958 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 09:26:41.439908 24958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/minicluster-data/ts-1/wal
uuid: "a740d9f90077438fa30c153b8967c788"
format_stamp: "Formatted at 2026-06-20 09:26:41 on dist-test-slave-706q"
server_key: "d521f39970ce9f1080450264ad0b9d32"
server_key_iv: "d7f77a60fa1a445d8e411acc2c28868d"
server_key_version: "encryptionkey@0"
I20260620 09:26:41.440009 24958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:26:41.449004 24958 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:26:41.451077 24958 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:26:41.451251 24958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:26:41.451493 24958 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:26:41.451841 24958 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:26:41.451897 24958 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:41.451949 24958 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:26:41.451979 24958 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:41.458001 24958 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.130:37323
I20260620 09:26:41.458094 25086 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.130:37323 every 8 connection(s)
I20260620 09:26:41.458439 24958 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/minicluster-data/ts-1/data/info.pb
I20260620 09:26:41.463681 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 24958
I20260620 09:26:41.463878 16558 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/minicluster-data/ts-1/wal/instance
I20260620 09:26:41.464100 16558 external_mini_cluster.cc:1428] Setting key ff0bd9b35ae4b53aaa6f284e8721b718
Jun 20 09:26:41 dist-test-slave-706q krb5kdc[24717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947601, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.130@KRBTEST.COM for kudu/127.16.43.190@KRBTEST.COM
I20260620 09:26:41.469362 24802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:41.459216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.130:59987 (local address 127.16.43.190:36415)
0620 09:26:41.459266 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:41.459267 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:41.459464 (+ 197us) server_negotiation.cc:407] Connection header received
0620 09:26:41.460018 (+ 554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:41.460019 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:41.460028 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:41.460091 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:41.461008 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:41.461499 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:41.462570 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:41.462747 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:41.462797 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:41.465615 (+ 2818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:41.465616 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:41.465616 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:26:41.465623 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:41.467503 (+ 1880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:41.467983 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:41.467984 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:41.467985 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:41.468052 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:41.468407 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:41.468408 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:41.468408 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:41.468565 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:41.468653 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:41.469154 (+ 501us) server_negotiation.cc:299] Negotiation successful
0620 09:26:41.469254 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 09:26:41.469448 25089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:41.459162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:36415 (local address 127.16.43.130:59987)
0620 09:26:41.459343 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:41.459387 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:41.459930 (+ 543us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:41.460152 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:41.460156 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:41.460728 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:41.460928 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:41.460933 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:41.461622 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:41.461623 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:41.462493 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:41.462496 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:41.462866 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:41.462867 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:41.462977 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:41.463592 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:26:41.463604 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:26:41.465503 (+ 1899us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:26:41.467623 (+ 2120us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:41.467630 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:41.467633 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:41.467900 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:41.468144 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:41.468145 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:41.468145 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:41.468331 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:41.468693 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:26:41.468699 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:26:41.468964 (+ 265us) client_negotiation.cc:769] Sending connection context
0620 09:26:41.469105 (+ 141us) client_negotiation.cc:240] Negotiation successful
0620 09:26:41.469242 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":99,"threads_started":1}
I20260620 09:26:41.470167 25087 heartbeater.cc:347] Connected to a master server at 127.16.43.190:36415
I20260620 09:26:41.470314 25087 heartbeater.cc:464] Registering TS with master...
I20260620 09:26:41.470530 25087 heartbeater.cc:511] Master 127.16.43.190:36415 requested a full tablet report, sending...
I20260620 09:26:41.471027 24762 ts_manager.cc:194] Registered new tserver with Master: a740d9f90077438fa30c153b8967c788 (127.16.43.130:37323)
I20260620 09:26:41.471585 24762 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.43.130@KRBTEST.COM'} at 127.16.43.130:59987
WARNING: no policy specified for kudu/127.16.43.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.43.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.43.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.131.keytab.
Entry for principal kudu/127.16.43.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.131.keytab.
I20260620 09:26:41.491307 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.43.131:0
--local_ip_for_outbound_sockets=127.16.43.131
--webserver_interface=127.16.43.131
--webserver_port=0
--tserver_master_addrs=127.16.43.190:36415
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.131.keytab
--principal=kudu/127.16.43.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.43.148:37741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:26:41.572052 25092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:26:41.572259 25092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:26:41.572283 25092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:26:41.573475 25092 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:26:41.573518 25092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:26:41.573546 25092 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.43.131
I20260620 09:26:41.575068 25092 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:37741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.131.keytab
--principal=kudu/127.16.43.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.43.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.43.131
--webserver_port=0
--tserver_master_addrs=127.16.43.190:36415
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25092
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.43.131
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:26:41.575296 25092 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:26:41.575567 25092 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:26:41.580641 25097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:26:41.580668 25100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:26:41.580721 25098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:26:41.580758 25092 server_base.cc:1034] running on GCE node
I20260620 09:26:41.581185 25092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:26:41.581427 25092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:26:41.582590 25092 hybrid_clock.cc:648] HybridClock initialized: now 1781947601582584 us; error 38 us; skew 500 ppm
Jun 20 09:26:41 dist-test-slave-706q krb5kdc[24717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947601, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:26:41.585315 25092 init.cc:377] Logged in from keytab as kudu/127.16.43.131@KRBTEST.COM (short username kudu)
I20260620 09:26:41.585789 25092 webserver.cc:460] Webserver started at http://127.16.43.131:41911/ using document root <none> and password file <none>
I20260620 09:26:41.586043 25092 fs_manager.cc:359] Metadata directory not provided
I20260620 09:26:41.586115 25092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:26:41.586261 25092 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:26:41.587320 25092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/minicluster-data/ts-2/data/instance:
uuid: "f449f1fb687248b7b700a9a780f530fd"
format_stamp: "Formatted at 2026-06-20 09:26:41 on dist-test-slave-706q"
server_key: "e94470ac038daec8b5ab68f21a0562ec"
server_key_iv: "fb2846d8082b58e430323257d38c1a70"
server_key_version: "encryptionkey@0"
I20260620 09:26:41.587710 25092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/minicluster-data/ts-2/wal/instance:
uuid: "f449f1fb687248b7b700a9a780f530fd"
format_stamp: "Formatted at 2026-06-20 09:26:41 on dist-test-slave-706q"
server_key: "e94470ac038daec8b5ab68f21a0562ec"
server_key_iv: "fb2846d8082b58e430323257d38c1a70"
server_key_version: "encryptionkey@0"
I20260620 09:26:41.589071 25092 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 09:26:41.589886 25107 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:41.590081 25092 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 09:26:41.590188 25092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/minicluster-data/ts-2/wal
uuid: "f449f1fb687248b7b700a9a780f530fd"
format_stamp: "Formatted at 2026-06-20 09:26:41 on dist-test-slave-706q"
server_key: "e94470ac038daec8b5ab68f21a0562ec"
server_key_iv: "fb2846d8082b58e430323257d38c1a70"
server_key_version: "encryptionkey@0"
I20260620 09:26:41.590260 25092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:26:41.597878 25092 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:26:41.600247 25092 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:26:41.600418 25092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:26:41.600660 25092 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:26:41.600978 25092 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:26:41.601053 25092 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:41.601094 25092 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:26:41.601122 25092 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:41.607017 25092 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.131:45623
I20260620 09:26:41.607064 25220 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.131:45623 every 8 connection(s)
I20260620 09:26:41.607441 25092 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/minicluster-data/ts-2/data/info.pb
Jun 20 09:26:41 dist-test-slave-706q krb5kdc[24717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947601, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.131@KRBTEST.COM for kudu/127.16.43.190@KRBTEST.COM
I20260620 09:26:41.616326 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 25092
I20260620 09:26:41.616444 16558 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/minicluster-data/ts-2/wal/instance
I20260620 09:26:41.616667 16558 external_mini_cluster.cc:1428] Setting key c36e5a8629a784e29f8142d8302f48c6
I20260620 09:26:41.617774 24802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:41.608314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.131:33227 (local address 127.16.43.190:36415)
0620 09:26:41.608383 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:41.608385 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:41.608644 (+ 259us) server_negotiation.cc:407] Connection header received
0620 09:26:41.609308 (+ 664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:41.609309 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:41.609321 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:41.609377 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:41.610331 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:41.610839 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:41.611921 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:41.612137 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:41.612225 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:41.614461 (+ 2236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:41.614462 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:41.614462 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:26:41.614469 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:41.615963 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:41.616582 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:41.616584 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:41.616584 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:41.616660 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:41.617018 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:41.617019 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:41.617019 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:41.617181 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:41.617254 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:41.617612 (+ 358us) server_negotiation.cc:299] Negotiation successful
0620 09:26:41.617708 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 09:26:41.618012 25223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:41.608301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:36415 (local address 127.16.43.131:33227)
0620 09:26:41.608515 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:41.608565 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:41.609199 (+ 634us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:41.609459 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:41.609462 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:41.610037 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:41.610232 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:41.610236 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:41.610943 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:41.610944 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:41.611820 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:41.611822 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:41.612234 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:41.612235 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:41.612363 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:41.612919 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:26:41.612928 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:26:41.614364 (+ 1436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:26:41.616043 (+ 1679us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:41.616049 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:41.616050 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:41.616424 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:41.616758 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:26:41.616759 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:26:41.616760 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:26:41.616913 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:26:41.617252 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:26:41.617259 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:26:41.617440 (+ 181us) client_negotiation.cc:769] Sending connection context
0620 09:26:41.617576 (+ 136us) client_negotiation.cc:240] Negotiation successful
0620 09:26:41.617754 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":129,"threads_started":1}
I20260620 09:26:41.618460 25221 heartbeater.cc:347] Connected to a master server at 127.16.43.190:36415
I20260620 09:26:41.618571 25221 heartbeater.cc:464] Registering TS with master...
I20260620 09:26:41.618754 25221 heartbeater.cc:511] Master 127.16.43.190:36415 requested a full tablet report, sending...
I20260620 09:26:41.619124 24762 ts_manager.cc:194] Registered new tserver with Master: f449f1fb687248b7b700a9a780f530fd (127.16.43.131:45623)
I20260620 09:26:41.619521 24762 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.43.131@KRBTEST.COM'} at 127.16.43.131:33227
I20260620 09:26:41.628443 16558 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.16.43.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.43.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.43.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/krb5kdc/oryx_127.16.43.129.keytab.
Entry for principal oryx/127.16.43.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/krb5kdc/oryx_127.16.43.129.keytab.
WARNING: no policy specified for oryx/127.16.43.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.43.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.43.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/krb5kdc/oryx_127.16.43.130.keytab.
Entry for principal oryx/127.16.43.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/krb5kdc/oryx_127.16.43.130.keytab.
WARNING: no policy specified for oryx/127.16.43.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.43.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.43.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/krb5kdc/oryx_127.16.43.131.keytab.
Entry for principal oryx/127.16.43.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/krb5kdc/oryx_127.16.43.131.keytab.
I20260620 09:26:41.709993 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 24824
I20260620 09:26:41.715282 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 24958
I20260620 09:26:41.719694 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 25092
I20260620 09:26:41.724251 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 24730
I20260620 09:26:41.729660 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.43.190:36415
--webserver_interface=127.16.43.190
--webserver_port=38249
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.190.keytab
--principal=kudu/127.16.43.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.43.148:37741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.43.190:36415
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:26:41.811029 25230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:26:41.811244 25230 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:26:41.811265 25230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:26:41.812475 25230 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:26:41.812520 25230 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:26:41.812533 25230 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:26:41.812546 25230 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:26:41.812559 25230 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:26:41.814083 25230 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:37741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.43.190:36415
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.190.keytab
--principal=kudu/127.16.43.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.43.190:36415
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.43.190
--webserver_port=38249
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25230
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:26:41.814348 25230 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:26:41.814599 25230 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:26:41.819667 25236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:26:41.819703 25235 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:26:41.819679 25238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:26:41.819916 25230 server_base.cc:1034] running on GCE node
I20260620 09:26:41.820178 25230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:26:41.820479 25230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:26:41.821624 25230 hybrid_clock.cc:648] HybridClock initialized: now 1781947601821614 us; error 28 us; skew 500 ppm
Jun 20 09:26:41 dist-test-slave-706q krb5kdc[24717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947601, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:26:41.824282 25230 init.cc:377] Logged in from keytab as kudu/127.16.43.190@KRBTEST.COM (short username kudu)
I20260620 09:26:41.824720 25230 webserver.cc:460] Webserver started at http://127.16.43.190:38249/ using document root <none> and password file <none>
I20260620 09:26:41.824926 25230 fs_manager.cc:359] Metadata directory not provided
I20260620 09:26:41.824983 25230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:26:41.826758 25230 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 09:26:41.827467 25245 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:41.827673 25230 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:26:41.827749 25230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/minicluster-data/master-0/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/minicluster-data/master-0/wal
uuid: "de251ca09785457c9c81526d5bcde08a"
format_stamp: "Formatted at 2026-06-20 09:26:41 on dist-test-slave-706q"
server_key: "2a9c613956e2bc4c1849c09a15861e35"
server_key_iv: "c5ba19957e07378564059d09eaa5671d"
server_key_version: "encryptionkey@0"
I20260620 09:26:41.828092 25230 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:26:41.837852 25230 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:26:41.840337 25230 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:26:41.840458 25230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:26:41.844554 25230 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.190:36415
I20260620 09:26:41.844830 25297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.190:36415 every 8 connection(s)
I20260620 09:26:41.845064 25230 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
I20260620 09:26:41.846314 25298 sys_catalog.cc:263] Verifying existing consensus state
I20260620 09:26:41.846964 25298 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de251ca09785457c9c81526d5bcde08a: Bootstrap starting.
I20260620 09:26:41.854919 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 25230
I20260620 09:26:41.855183 25298 log.cc:824] T 00000000000000000000000000000000 P de251ca09785457c9c81526d5bcde08a: Log is configured to *not* fsync() on all Append() calls
I20260620 09:26:41.855233 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.43.129:44231
--local_ip_for_outbound_sockets=127.16.43.129
--tserver_master_addrs=127.16.43.190:36415
--webserver_port=34293
--webserver_interface=127.16.43.129
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.129.keytab
--principal=kudu/127.16.43.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.43.148:37741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.16.43.129
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/krb5kdc/oryx_127.16.43.129.keytab with env {KRB5CCNAME=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 09:26:41.856913 25298 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de251ca09785457c9c81526d5bcde08a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 09:26:41.857225 25298 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de251ca09785457c9c81526d5bcde08a: Bootstrap complete.
I20260620 09:26:41.859433 25298 raft_consensus.cc:348] T 00000000000000000000000000000000 P de251ca09785457c9c81526d5bcde08a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de251ca09785457c9c81526d5bcde08a" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 36415 } }
I20260620 09:26:41.859747 25298 raft_consensus.cc:729] T 00000000000000000000000000000000 P de251ca09785457c9c81526d5bcde08a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: de251ca09785457c9c81526d5bcde08a, State: Initialized, Role: FOLLOWER
I20260620 09:26:41.859941 25298 consensus_queue.cc:260] T 00000000000000000000000000000000 P de251ca09785457c9c81526d5bcde08a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de251ca09785457c9c81526d5bcde08a" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 36415 } }
I20260620 09:26:41.860018 25298 raft_consensus.cc:388] T 00000000000000000000000000000000 P de251ca09785457c9c81526d5bcde08a [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:26:41.860071 25298 raft_consensus.cc:482] T 00000000000000000000000000000000 P de251ca09785457c9c81526d5bcde08a [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:26:41.860121 25298 raft_consensus.cc:3037] T 00000000000000000000000000000000 P de251ca09785457c9c81526d5bcde08a [term 1 FOLLOWER]: Advancing to term 2
I20260620 09:26:41.861109 25298 raft_consensus.cc:504] T 00000000000000000000000000000000 P de251ca09785457c9c81526d5bcde08a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de251ca09785457c9c81526d5bcde08a" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 36415 } }
I20260620 09:26:41.861234 25298 leader_election.cc:302] T 00000000000000000000000000000000 P de251ca09785457c9c81526d5bcde08a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: de251ca09785457c9c81526d5bcde08a; no voters:
I20260620 09:26:41.861451 25298 leader_election.cc:288] T 00000000000000000000000000000000 P de251ca09785457c9c81526d5bcde08a [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 09:26:41.861490 25302 raft_consensus.cc:2781] T 00000000000000000000000000000000 P de251ca09785457c9c81526d5bcde08a [term 2 FOLLOWER]: Leader election won for term 2
I20260620 09:26:41.861599 25302 raft_consensus.cc:686] T 00000000000000000000000000000000 P de251ca09785457c9c81526d5bcde08a [term 2 LEADER]: Becoming Leader. State: Replica: de251ca09785457c9c81526d5bcde08a, State: Running, Role: LEADER
I20260620 09:26:41.861683 25302 consensus_queue.cc:237] T 00000000000000000000000000000000 P de251ca09785457c9c81526d5bcde08a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de251ca09785457c9c81526d5bcde08a" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 36415 } }
I20260620 09:26:41.861716 25298 sys_catalog.cc:564] T 00000000000000000000000000000000 P de251ca09785457c9c81526d5bcde08a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:26:41.861949 25302 sys_catalog.cc:455] T 00000000000000000000000000000000 P de251ca09785457c9c81526d5bcde08a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "de251ca09785457c9c81526d5bcde08a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de251ca09785457c9c81526d5bcde08a" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 36415 } } }
I20260620 09:26:41.862095 25302 sys_catalog.cc:458] T 00000000000000000000000000000000 P de251ca09785457c9c81526d5bcde08a [sys.catalog]: This master's current role is: LEADER
I20260620 09:26:41.862272 25302 sys_catalog.cc:455] T 00000000000000000000000000000000 P de251ca09785457c9c81526d5bcde08a [sys.catalog]: SysCatalogTable state changed. Reason: New leader de251ca09785457c9c81526d5bcde08a. Latest consensus state: current_term: 2 leader_uuid: "de251ca09785457c9c81526d5bcde08a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de251ca09785457c9c81526d5bcde08a" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 36415 } } }
I20260620 09:26:41.862340 25308 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:26:41.862416 25302 sys_catalog.cc:458] T 00000000000000000000000000000000 P de251ca09785457c9c81526d5bcde08a [sys.catalog]: This master's current role is: LEADER
I20260620 09:26:41.863086 25308 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:26:41.863492 25308 catalog_manager.cc:1259] Loaded cluster ID: 98a0d94d2db343ffa2b8c2ab8defabc4
I20260620 09:26:41.863561 25308 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:26:41.864955 25308 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:26:41.865144 25308 catalog_manager.cc:5879] T 00000000000000000000000000000000 P de251ca09785457c9c81526d5bcde08a: Loaded TSK: 0
I20260620 09:26:41.865398 25308 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 09:26:41.940629 25301 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:26:41.940855 25301 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:26:41.940881 25301 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:26:41.942183 25301 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:26:41.942231 25301 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:26:41.942261 25301 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.43.129
I20260620 09:26:41.943854 25301 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:37741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/krb5kdc/oryx_127.16.43.129.keytab
--principal=oryx/127.16.43.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.43.129:44231
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.43.129
--webserver_port=34293
--tserver_master_addrs=127.16.43.190:36415
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25301
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.43.129
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:26:41.944099 25301 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:26:41.944380 25301 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:26:41.949574 25324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:26:41.949683 25301 server_base.cc:1034] running on GCE node
W20260620 09:26:41.949586 25327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:26:41.949571 25325 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:26:41.950188 25301 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:26:41.950467 25301 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:26:41.951601 25301 hybrid_clock.cc:648] HybridClock initialized: now 1781947601951588 us; error 31 us; skew 500 ppm
Jun 20 09:26:41 dist-test-slave-706q krb5kdc[24717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947601, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.43.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:26:41.954381 25301 init.cc:377] Logged in from keytab as oryx/127.16.43.129@KRBTEST.COM (short username oryx)
I20260620 09:26:41.954852 25301 webserver.cc:460] Webserver started at http://127.16.43.129:34293/ using document root <none> and password file <none>
I20260620 09:26:41.955081 25301 fs_manager.cc:359] Metadata directory not provided
I20260620 09:26:41.955163 25301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:26:41.956712 25301 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:26:41.957419 25334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:41.957665 25301 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:26:41.957783 25301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/minicluster-data/ts-0/wal
uuid: "f37759d2ae5a4d1a85dfa10108c5092c"
format_stamp: "Formatted at 2026-06-20 09:26:41 on dist-test-slave-706q"
server_key: "bf3760c9ddc85f105f27d191a5034f61"
server_key_iv: "0d51c6552c24da6684c3ae06c9f0a233"
server_key_version: "encryptionkey@0"
I20260620 09:26:41.958184 25301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:26:41.973578 25301 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:26:41.976362 25301 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:26:41.976560 25301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:26:41.976833 25301 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:26:41.977175 25301 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:26:41.977243 25301 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:41.977294 25301 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:26:41.977320 25301 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:41.984473 25301 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.129:44231
I20260620 09:26:41.984519 25447 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.129:44231 every 8 connection(s)
I20260620 09:26:41.984946 25301 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/minicluster-data/ts-0/data/info.pb
I20260620 09:26:41.990494 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 25301
I20260620 09:26:41.990831 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.43.130:37323
--local_ip_for_outbound_sockets=127.16.43.130
--tserver_master_addrs=127.16.43.190:36415
--webserver_port=42213
--webserver_interface=127.16.43.130
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.130.keytab
--principal=kudu/127.16.43.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.43.148:37741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.16.43.130
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/krb5kdc/oryx_127.16.43.130.keytab with env {KRB5CCNAME=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 09:26:41 dist-test-slave-706q krb5kdc[24717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.43.129@KRBTEST.COM for oryx/127.16.43.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:26:41 dist-test-slave-706q krb5kdc[24717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.43.129@KRBTEST.COM for oryx/127.16.43.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:26:41.995935 25450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:41.985719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:36415 (local address 127.16.43.129:36863)
0620 09:26:41.985939 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:41.985998 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:41.986612 (+ 614us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:41.987329 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:41.987333 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:41.988019 (+ 686us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:41.988218 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:41.988223 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:41.989871 (+ 1648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:41.989871 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:41.991086 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:41.991089 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:41.992014 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:41.992015 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:41.992157 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:41.992716 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:26:41.992726 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:26:41.995659 (+ 2933us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.43.190:36415: Server oryx/127.16.43.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":92,"threads_started":1}
W20260620 09:26:41.995968 25451 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:26:41.985842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.129:36863 (local address 127.16.43.190:36415)
0620 09:26:41.986905 (+ 1063us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:41.986914 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:41.986948 (+ 34us) server_negotiation.cc:407] Connection header received
0620 09:26:41.987066 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:41.987071 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:41.987127 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:41.987257 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:41.988345 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:41.989759 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:41.991191 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:41.991905 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:41.992019 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:41.995775 (+ 3756us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.43.129:36863: BlockingRecv error: recv got EOF from 127.16.43.129:36863 (error 108)
Metrics: {"server-negotiator.queue_time_us":919,"thread_start_us":88,"threads_started":1}
W20260620 09:26:41.996213 25448 heartbeater.cc:644] Failed to heartbeat to 127.16.43.190:36415 (0 consecutive failures): Not authorized: Failed to ping master at 127.16.43.190:36415: Client connection negotiation failed: client connection to 127.16.43.190:36415: Server oryx/127.16.43.190@KRBTEST.COM not found in Kerberos database
Jun 20 09:26:41 dist-test-slave-706q krb5kdc[24717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.43.129@KRBTEST.COM for oryx/127.16.43.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:26:41 dist-test-slave-706q krb5kdc[24717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.43.129@KRBTEST.COM for oryx/127.16.43.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:26:42.001186 25450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:41.996462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:36415 (local address 127.16.43.129:42547)
0620 09:26:41.996581 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:41.996592 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:41.996646 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:41.996776 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:41.996777 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:41.996915 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:41.996967 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:41.996968 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:41.997622 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:41.997623 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:41.998124 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:41.998126 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:41.998512 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:41.998513 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:41.998600 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:41.999154 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:26:41.999159 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:26:42.001134 (+ 1975us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.43.190:36415: Server oryx/127.16.43.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":100}
W20260620 09:26:42.001348 25451 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:26:41.996535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.129:42547 (local address 127.16.43.190:36415)
0620 09:26:41.996574 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:41.996576 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:41.996630 (+ 54us) server_negotiation.cc:407] Connection header received
0620 09:26:41.996707 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:41.996708 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:41.996719 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:41.996766 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:41.997045 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:41.997526 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:41.998211 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:41.998424 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:41.998511 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:42.001263 (+ 2752us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.43.129:42547: BlockingRecv error: recv got EOF from 127.16.43.129:42547 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
Jun 20 09:26:42 dist-test-slave-706q krb5kdc[24717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.43.129@KRBTEST.COM for oryx/127.16.43.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:26:42 dist-test-slave-706q krb5kdc[24717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.43.129@KRBTEST.COM for oryx/127.16.43.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:26:42.006718 25453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:42.001453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:36415 (local address 127.16.43.129:53807)
0620 09:26:42.001620 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:42.001633 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:42.001755 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:42.001913 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:42.001914 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:42.002125 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:42.002199 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:42.002201 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:42.002810 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:42.002811 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:42.003759 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:42.003762 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:42.004120 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:42.004121 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:42.004186 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:42.004667 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:26:42.004672 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:26:42.006631 (+ 1959us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.43.190:36415: Server oryx/127.16.43.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":84,"threads_started":1}
W20260620 09:26:42.006852 25448 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 09:26:42.006851 25451 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:26:42.001628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.129:53807 (local address 127.16.43.190:36415)
0620 09:26:42.001672 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:42.001674 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:42.001769 (+ 95us) server_negotiation.cc:407] Connection header received
0620 09:26:42.001817 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:42.001818 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:42.001830 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:42.001883 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:42.002291 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:42.002731 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:42.003847 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:42.004028 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:42.004084 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:42.006753 (+ 2669us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.43.129:53807: BlockingRecv error: recv got EOF from 127.16.43.129:53807 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260620 09:26:42.077883 25452 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:26:42.078114 25452 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:26:42.078140 25452 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:26:42.079265 25452 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:26:42.079308 25452 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:26:42.079336 25452 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.43.130
I20260620 09:26:42.080729 25452 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:37741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/krb5kdc/oryx_127.16.43.130.keytab
--principal=oryx/127.16.43.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.43.130:37323
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.43.130
--webserver_port=42213
--tserver_master_addrs=127.16.43.190:36415
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25452
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.43.130
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:26:42.080925 25452 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:26:42.081106 25452 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:26:42.086582 25459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:26:42.086577 25458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:26:42.086577 25461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:26:42.086665 25452 server_base.cc:1034] running on GCE node
I20260620 09:26:42.087002 25452 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:26:42.087288 25452 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:26:42.088474 25452 hybrid_clock.cc:648] HybridClock initialized: now 1781947602088461 us; error 46 us; skew 500 ppm
Jun 20 09:26:42 dist-test-slave-706q krb5kdc[24717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947602, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.43.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:26:42.091037 25452 init.cc:377] Logged in from keytab as oryx/127.16.43.130@KRBTEST.COM (short username oryx)
I20260620 09:26:42.091512 25452 webserver.cc:460] Webserver started at http://127.16.43.130:42213/ using document root <none> and password file <none>
I20260620 09:26:42.091792 25452 fs_manager.cc:359] Metadata directory not provided
I20260620 09:26:42.091850 25452 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:26:42.093209 25452 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 09:26:42.093901 25468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:42.094108 25452 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 09:26:42.094218 25452 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/minicluster-data/ts-1/wal
uuid: "a740d9f90077438fa30c153b8967c788"
format_stamp: "Formatted at 2026-06-20 09:26:41 on dist-test-slave-706q"
server_key: "d521f39970ce9f1080450264ad0b9d32"
server_key_iv: "d7f77a60fa1a445d8e411acc2c28868d"
server_key_version: "encryptionkey@0"
I20260620 09:26:42.094576 25452 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:26:42.109169 25452 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:26:42.111403 25452 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:26:42.111617 25452 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:26:42.111873 25452 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:26:42.112196 25452 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:26:42.112255 25452 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:42.112303 25452 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:26:42.112326 25452 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:42.118203 25452 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.130:37323
I20260620 09:26:42.118276 25581 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.130:37323 every 8 connection(s)
I20260620 09:26:42.118727 25452 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/minicluster-data/ts-1/data/info.pb
Jun 20 09:26:42 dist-test-slave-706q krb5kdc[24717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.43.130@KRBTEST.COM for oryx/127.16.43.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:26:42 dist-test-slave-706q krb5kdc[24717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.43.130@KRBTEST.COM for oryx/127.16.43.190@KRBTEST.COM, Server not found in Kerberos database
W20260620 09:26:42.126925 25451 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:26:42.119786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.130:40357 (local address 127.16.43.190:36415)
0620 09:26:42.119842 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:42.119844 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:42.120142 (+ 298us) server_negotiation.cc:407] Connection header received
0620 09:26:42.120948 (+ 806us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:42.120949 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:42.120965 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:42.121054 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:42.122211 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:42.122714 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:42.123764 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:42.123957 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:42.124014 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:42.126856 (+ 2842us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.43.130:40357: BlockingRecv error: recv got EOF from 127.16.43.130:40357 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 09:26:42.126982 25584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:42.119738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:36415 (local address 127.16.43.130:40357)
0620 09:26:42.119965 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:42.120051 (+ 86us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:42.120787 (+ 736us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:42.121153 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:42.121159 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:42.121912 (+ 753us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:42.122111 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:42.122115 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:42.122797 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:42.122797 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:42.123691 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:42.123693 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:42.124030 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:42.124030 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:42.124126 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:42.124700 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:26:42.124709 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:26:42.126767 (+ 2058us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.43.190:36415: Server oryx/127.16.43.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":88,"threads_started":1}
I20260620 09:26:42.127113 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 25452
I20260620 09:26:42.127398 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.43.131:45623
--local_ip_for_outbound_sockets=127.16.43.131
--tserver_master_addrs=127.16.43.190:36415
--webserver_port=41911
--webserver_interface=127.16.43.131
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.131.keytab
--principal=kudu/127.16.43.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.43.148:37741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.16.43.131
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/krb5kdc/oryx_127.16.43.131.keytab with env {KRB5CCNAME=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:26:42.127758 25582 heartbeater.cc:644] Failed to heartbeat to 127.16.43.190:36415 (0 consecutive failures): Not authorized: Failed to ping master at 127.16.43.190:36415: Client connection negotiation failed: client connection to 127.16.43.190:36415: Server oryx/127.16.43.190@KRBTEST.COM not found in Kerberos database
Jun 20 09:26:42 dist-test-slave-706q krb5kdc[24717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.43.130@KRBTEST.COM for oryx/127.16.43.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:26:42 dist-test-slave-706q krb5kdc[24717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.43.130@KRBTEST.COM for oryx/127.16.43.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:26:42.133152 25584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:42.128001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:36415 (local address 127.16.43.130:42123)
0620 09:26:42.128051 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:42.128064 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:42.128126 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:42.128264 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:42.128265 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:42.128395 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:42.128438 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:42.128439 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:42.129359 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:42.129360 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:42.130044 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:42.130046 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:42.130641 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:42.130642 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:42.130699 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:42.131240 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:26:42.131245 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:26:42.133101 (+ 1856us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.43.190:36415: Server oryx/127.16.43.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":33}
W20260620 09:26:42.133303 25451 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:26:42.128066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.130:42123 (local address 127.16.43.190:36415)
0620 09:26:42.128130 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:42.128132 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:42.128141 (+ 9us) server_negotiation.cc:407] Connection header received
0620 09:26:42.128185 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:42.128186 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:42.128195 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:42.128261 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:42.128515 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:42.129268 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:42.130185 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:42.130512 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:42.130583 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:42.133219 (+ 2636us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.43.130:42123: BlockingRecv error: recv got EOF from 127.16.43.130:42123 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Jun 20 09:26:42 dist-test-slave-706q krb5kdc[24717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.43.130@KRBTEST.COM for oryx/127.16.43.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:26:42 dist-test-slave-706q krb5kdc[24717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.43.130@KRBTEST.COM for oryx/127.16.43.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:26:42.138586 25586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:42.133421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:36415 (local address 127.16.43.130:34263)
0620 09:26:42.133590 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:42.133600 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:42.133661 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:42.133858 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:42.133859 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:42.133988 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:42.134035 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:42.134037 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:42.134992 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:42.134993 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:42.135619 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:42.135621 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:42.136150 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:42.136151 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:42.136200 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:42.136714 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:26:42.136719 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:26:42.138497 (+ 1778us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.43.190:36415: Server oryx/127.16.43.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":115,"threads_started":1}
W20260620 09:26:42.138721 25451 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:26:42.133484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.130:34263 (local address 127.16.43.190:36415)
0620 09:26:42.133528 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:42.133530 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:42.133677 (+ 147us) server_negotiation.cc:407] Connection header received
0620 09:26:42.133707 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:42.133708 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:42.133720 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:42.133816 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:42.134106 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:42.134901 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:42.135698 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:42.136026 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:42.136108 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:42.138604 (+ 2496us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.43.130:34263: BlockingRecv error: recv got EOF from 127.16.43.130:34263 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
W20260620 09:26:42.138844 25582 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 09:26:42.212024 25585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:26:42.212240 25585 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:26:42.212270 25585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:26:42.213619 25585 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:26:42.213689 25585 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:26:42.213722 25585 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.43.131
I20260620 09:26:42.215166 25585 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:37741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/krb5kdc/oryx_127.16.43.131.keytab
--principal=oryx/127.16.43.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.43.131:45623
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.43.131
--webserver_port=41911
--tserver_master_addrs=127.16.43.190:36415
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25585
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.43.131
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:26:42.215420 25585 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:26:42.215687 25585 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:26:42.220793 25592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:26:42.220779 25594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:26:42.220779 25591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:26:42.221354 25585 server_base.cc:1034] running on GCE node
I20260620 09:26:42.221580 25585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:26:42.221819 25585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:26:42.222975 25585 hybrid_clock.cc:648] HybridClock initialized: now 1781947602222962 us; error 29 us; skew 500 ppm
Jun 20 09:26:42 dist-test-slave-706q krb5kdc[24717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947602, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.43.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:26:42.225556 25585 init.cc:377] Logged in from keytab as oryx/127.16.43.131@KRBTEST.COM (short username oryx)
I20260620 09:26:42.226092 25585 webserver.cc:460] Webserver started at http://127.16.43.131:41911/ using document root <none> and password file <none>
I20260620 09:26:42.226336 25585 fs_manager.cc:359] Metadata directory not provided
I20260620 09:26:42.226419 25585 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:26:42.227836 25585 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:26:42.228415 25601 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:42.228612 25585 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:42.228719 25585 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/minicluster-data/ts-2/wal
uuid: "f449f1fb687248b7b700a9a780f530fd"
format_stamp: "Formatted at 2026-06-20 09:26:41 on dist-test-slave-706q"
server_key: "e94470ac038daec8b5ab68f21a0562ec"
server_key_iv: "fb2846d8082b58e430323257d38c1a70"
server_key_version: "encryptionkey@0"
I20260620 09:26:42.229091 25585 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:26:42.240592 25585 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:26:42.242980 25585 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:26:42.243151 25585 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:26:42.243382 25585 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:26:42.243775 25585 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:26:42.243829 25585 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:42.243872 25585 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:26:42.243901 25585 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:42.250027 25585 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.131:45623
I20260620 09:26:42.250082 25714 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.131:45623 every 8 connection(s)
I20260620 09:26:42.250458 25585 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781947562949448-16558-0/minicluster-data/ts-2/data/info.pb
I20260620 09:26:42.253252 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 25585
Jun 20 09:26:42 dist-test-slave-706q krb5kdc[24717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.43.131@KRBTEST.COM for oryx/127.16.43.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:26:42 dist-test-slave-706q krb5kdc[24717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.43.131@KRBTEST.COM for oryx/127.16.43.190@KRBTEST.COM, Server not found in Kerberos database
W20260620 09:26:42.259131 25451 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:26:42.251275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.131:34853 (local address 127.16.43.190:36415)
0620 09:26:42.251322 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:42.251324 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:42.251570 (+ 246us) server_negotiation.cc:407] Connection header received
0620 09:26:42.252068 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:42.252070 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:42.252081 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:42.252135 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:42.253075 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:42.253736 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:42.255230 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:42.255577 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:42.255638 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:42.259059 (+ 3421us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.43.131:34853: BlockingRecv error: recv got EOF from 127.16.43.131:34853 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 09:26:42.259160 25717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:42.251296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:36415 (local address 127.16.43.131:34853)
0620 09:26:42.251448 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:42.251484 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:42.251978 (+ 494us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:42.252188 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:42.252191 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:42.252762 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:42.252963 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:42.252968 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:42.253829 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:42.253830 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:42.255146 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:42.255150 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:42.255648 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:42.255649 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:42.255787 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:42.256351 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:26:42.256362 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:26:42.258956 (+ 2594us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.43.190:36415: Server oryx/127.16.43.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":87,"threads_started":1}
W20260620 09:26:42.259409 25715 heartbeater.cc:644] Failed to heartbeat to 127.16.43.190:36415 (0 consecutive failures): Not authorized: Failed to ping master at 127.16.43.190:36415: Client connection negotiation failed: client connection to 127.16.43.190:36415: Server oryx/127.16.43.190@KRBTEST.COM not found in Kerberos database
I20260620 09:26:42.262097 25719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:42.253791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54358 (local address 127.16.43.190:36415)
0620 09:26:42.253895 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:42.253896 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:42.253902 (+ 6us) server_negotiation.cc:407] Connection header received
0620 09:26:42.253920 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:42.253921 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:42.253931 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:42.254000 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:42.254822 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:42.255382 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:42.256442 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:42.256641 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:42.256691 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:42.257990 (+ 1299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:42.257995 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:42.257996 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:26:42.258009 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:42.260366 (+ 2357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:42.260755 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:42.260759 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:42.260759 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:42.260874 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:42.261106 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:42.261107 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:42.261107 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:42.261317 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:42.261415 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:42.261883 (+ 468us) server_negotiation.cc:299] Negotiation successful
0620 09:26:42.261995 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85,"thread_start_us":66,"threads_started":1}
Jun 20 09:26:42 dist-test-slave-706q krb5kdc[24717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.43.131@KRBTEST.COM for oryx/127.16.43.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:26:42 dist-test-slave-706q krb5kdc[24717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.43.131@KRBTEST.COM for oryx/127.16.43.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:26:42.264504 25717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:42.259671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:36415 (local address 127.16.43.131:47779)
0620 09:26:42.259725 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:42.259737 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:42.259782 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:42.259945 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:42.259945 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:42.260049 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:42.260093 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:42.260094 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:42.260977 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:42.260978 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:42.261744 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:42.261746 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:42.262278 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:42.262278 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:42.262344 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:42.262748 (+ 404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:26:42.262753 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:26:42.264450 (+ 1697us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.43.190:36415: Server oryx/127.16.43.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":36}
W20260620 09:26:42.264633 25451 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:26:42.259743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.131:47779 (local address 127.16.43.190:36415)
0620 09:26:42.259801 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:42.259803 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:42.259814 (+ 11us) server_negotiation.cc:407] Connection header received
0620 09:26:42.259834 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:42.259835 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:42.259844 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:42.259910 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:42.260190 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:42.260873 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:42.261834 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:42.262114 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:42.262160 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:42.264561 (+ 2401us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.43.131:47779: BlockingRecv error: recv got EOF from 127.16.43.131:47779 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Jun 20 09:26:42 dist-test-slave-706q krb5kdc[24717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.43.131@KRBTEST.COM for oryx/127.16.43.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:26:42 dist-test-slave-706q krb5kdc[24717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.43.131@KRBTEST.COM for oryx/127.16.43.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:26:42.268400 25717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:42.264750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:36415 (local address 127.16.43.131:38099)
0620 09:26:42.264826 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:42.264843 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:42.264879 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:42.264971 (+ 92us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:42.264971 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:42.265072 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:42.265098 (+ 26us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:42.265099 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:42.265657 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:42.265657 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:42.266125 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:42.266126 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:42.266446 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:42.266447 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:42.266489 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:42.266832 (+ 343us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:26:42.266836 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:26:42.268357 (+ 1521us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.43.190:36415: Server oryx/127.16.43.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":62}
W20260620 09:26:42.268473 25451 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:26:42.264789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.131:38099 (local address 127.16.43.190:36415)
0620 09:26:42.264821 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:42.264823 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:42.264869 (+ 46us) server_negotiation.cc:407] Connection header received
0620 09:26:42.264919 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:42.264919 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:42.264926 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:42.264969 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:42.265154 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:42.265591 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:42.266211 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:42.266379 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:42.266440 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:42.268437 (+ 1997us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.43.131:38099: BlockingRecv error: recv got EOF from 127.16.43.131:38099 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260620 09:26:42.268568 25715 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 20 09:26:43 dist-test-slave-706q krb5kdc[24717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.43.129@KRBTEST.COM for oryx/127.16.43.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:26:43 dist-test-slave-706q krb5kdc[24717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.43.129@KRBTEST.COM for oryx/127.16.43.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:26:43.012217 25720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:43.007266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:36415 (local address 127.16.43.129:36245)
0620 09:26:43.007441 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:43.007452 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:43.007506 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:43.007721 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:43.007722 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:43.007901 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:43.007952 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:43.007953 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:43.008638 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:43.008640 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:43.009234 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:43.009236 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:43.009638 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:43.009639 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:43.009681 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:43.010267 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:26:43.010273 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:26:43.012157 (+ 1884us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.43.190:36415: Server oryx/127.16.43.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":110,"threads_started":1}
W20260620 09:26:43.012403 25721 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:26:43.007344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.129:36245 (local address 127.16.43.190:36415)
0620 09:26:43.007491 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:43.007493 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:43.007502 (+ 9us) server_negotiation.cc:407] Connection header received
0620 09:26:43.007606 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:43.007607 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:43.007617 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:43.007682 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:43.008025 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:43.008549 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:43.009350 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:43.009566 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:43.009629 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:43.012308 (+ 2679us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.43.129:36245: BlockingRecv error: recv got EOF from 127.16.43.129:36245 (error 108)
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":88,"threads_started":1}
Jun 20 09:26:43 dist-test-slave-706q krb5kdc[24717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.43.130@KRBTEST.COM for oryx/127.16.43.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:26:43 dist-test-slave-706q krb5kdc[24717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.43.130@KRBTEST.COM for oryx/127.16.43.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:26:43.144467 25722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:43.139330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:36415 (local address 127.16.43.130:43725)
0620 09:26:43.139472 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:43.139489 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:43.139580 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:43.139801 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:43.139802 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:43.140037 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:43.140102 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:43.140105 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:43.140904 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:43.140904 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:43.141435 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:43.141436 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:43.141835 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:43.141836 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:43.141897 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:43.142432 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:26:43.142438 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:26:43.144406 (+ 1968us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.43.190:36415: Server oryx/127.16.43.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":89,"threads_started":1}
W20260620 09:26:43.144680 25721 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:26:43.139384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.130:43725 (local address 127.16.43.190:36415)
0620 09:26:43.139486 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:43.139488 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:43.139581 (+ 93us) server_negotiation.cc:407] Connection header received
0620 09:26:43.139670 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:43.139671 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:43.139686 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:43.139768 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:43.140196 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:43.140823 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:43.141548 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:43.141742 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:43.141810 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:43.144611 (+ 2801us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.43.130:43725: BlockingRecv error: recv got EOF from 127.16.43.130:43725 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Jun 20 09:26:43 dist-test-slave-706q krb5kdc[24717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.43.131@KRBTEST.COM for oryx/127.16.43.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:26:43 dist-test-slave-706q krb5kdc[24717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.43.131@KRBTEST.COM for oryx/127.16.43.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:26:43.274283 25723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:43.269199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:36415 (local address 127.16.43.131:53525)
0620 09:26:43.269388 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:43.269401 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:43.269467 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:43.269636 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:43.269637 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:43.269827 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:43.269877 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:43.269879 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:43.270559 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:43.270560 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:43.271118 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:43.271119 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:43.271497 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:43.271498 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:43.271594 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:43.272078 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:26:43.272083 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:26:43.274175 (+ 2092us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.43.190:36415: Server oryx/127.16.43.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":115,"threads_started":1}
W20260620 09:26:43.274320 25721 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:26:43.269403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.131:53525 (local address 127.16.43.190:36415)
0620 09:26:43.269462 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:43.269464 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:43.269475 (+ 11us) server_negotiation.cc:407] Connection header received
0620 09:26:43.269537 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:43.269538 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:43.269551 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:43.269627 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:43.269966 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:43.270464 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:43.271210 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:43.271419 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:43.271500 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:43.274252 (+ 2752us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.43.131:53525: BlockingRecv error: recv got EOF from 127.16.43.131:53525 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Jun 20 09:26:44 dist-test-slave-706q krb5kdc[24717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.43.129@KRBTEST.COM for oryx/127.16.43.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:26:44 dist-test-slave-706q krb5kdc[24717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.43.129@KRBTEST.COM for oryx/127.16.43.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:26:44.018636 25724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:44.012717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:36415 (local address 127.16.43.129:40559)
0620 09:26:44.012969 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:44.012980 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:44.013040 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:44.013216 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:44.013216 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:44.013377 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:44.013426 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:44.013428 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:44.014519 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:44.014521 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:44.015386 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:44.015389 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:44.015955 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:44.015956 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:44.016004 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:44.016522 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:26:44.016528 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:26:44.018507 (+ 1979us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.43.190:36415: Server oryx/127.16.43.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":168,"threads_started":1}
W20260620 09:26:44.019039 25725 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:26:44.012847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.129:40559 (local address 127.16.43.190:36415)
0620 09:26:44.012981 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:44.012984 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:44.013035 (+ 51us) server_negotiation.cc:407] Connection header received
0620 09:26:44.013098 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:44.013099 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:44.013115 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:44.013202 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:44.013509 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:44.014431 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:44.015469 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:44.015880 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:44.015957 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:44.018978 (+ 3021us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.43.129:40559: BlockingRecv error: recv got EOF from 127.16.43.129:40559 (error 108)
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":56,"threads_started":1}
Jun 20 09:26:44 dist-test-slave-706q krb5kdc[24717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.43.130@KRBTEST.COM for oryx/127.16.43.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:26:44 dist-test-slave-706q krb5kdc[24717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.43.130@KRBTEST.COM for oryx/127.16.43.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:26:44.150439 25726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:44.144951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:36415 (local address 127.16.43.130:60971)
0620 09:26:44.145187 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:44.145201 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:44.145315 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:44.145546 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:44.145548 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:44.145834 (+ 286us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:44.145890 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:44.145893 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:44.146873 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:44.146875 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:44.147579 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:44.147582 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:44.148033 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:44.148034 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:44.148085 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:44.148640 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:26:44.148646 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:26:44.150379 (+ 1733us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.43.190:36415: Server oryx/127.16.43.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":106,"threads_started":1}
W20260620 09:26:44.150588 25725 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:26:44.145067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.130:60971 (local address 127.16.43.190:36415)
0620 09:26:44.145166 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:44.145169 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:44.145290 (+ 121us) server_negotiation.cc:407] Connection header received
0620 09:26:44.145403 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:44.145404 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:44.145428 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:44.145539 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:44.146020 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:44.146769 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:44.147672 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:44.147933 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:44.148003 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:44.150508 (+ 2505us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.43.130:60971: BlockingRecv error: recv got EOF from 127.16.43.130:60971 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Jun 20 09:26:44 dist-test-slave-706q krb5kdc[24717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.43.131@KRBTEST.COM for oryx/127.16.43.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:26:44 dist-test-slave-706q krb5kdc[24717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.43.131@KRBTEST.COM for oryx/127.16.43.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:26:44.280318 25727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:44.274848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:36415 (local address 127.16.43.131:51335)
0620 09:26:44.274995 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:44.275016 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:44.275070 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:44.275296 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:44.275298 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:44.275495 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:44.275608 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:44.275611 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:44.276355 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:44.276356 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:44.276974 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:44.276977 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:44.277417 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:44.277418 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:44.277491 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:44.278001 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:26:44.278006 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:26:44.280258 (+ 2252us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.43.190:36415: Server oryx/127.16.43.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":87,"threads_started":1}
W20260620 09:26:44.280397 25725 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:26:44.275067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.131:51335 (local address 127.16.43.190:36415)
0620 09:26:44.275149 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:44.275151 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:44.275159 (+ 8us) server_negotiation.cc:407] Connection header received
0620 09:26:44.275200 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:44.275201 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:44.275214 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:44.275287 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:44.275705 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:44.276259 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:44.277103 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:44.277320 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:44.277416 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:44.280349 (+ 2933us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.43.131:51335: BlockingRecv error: recv got EOF from 127.16.43.131:51335 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Jun 20 09:26:45 dist-test-slave-706q krb5kdc[24717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.43.129@KRBTEST.COM for oryx/127.16.43.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:26:45 dist-test-slave-706q krb5kdc[24717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.43.129@KRBTEST.COM for oryx/127.16.43.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:26:45.024093 25728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:45.019317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:36415 (local address 127.16.43.129:60423)
0620 09:26:45.019591 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:45.019609 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:45.019658 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:45.019928 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:45.019928 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:45.020076 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:45.020126 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:45.020128 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:45.020767 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:45.020768 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:45.021340 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:45.021342 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:45.021664 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:45.021665 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:45.021705 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:45.022202 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:26:45.022206 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:26:45.024034 (+ 1828us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.43.190:36415: Server oryx/127.16.43.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":134,"threads_started":1}
W20260620 09:26:45.024264 25729 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:26:45.019664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.129:60423 (local address 127.16.43.190:36415)
0620 09:26:45.019798 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:45.019800 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:45.019808 (+ 8us) server_negotiation.cc:407] Connection header received
0620 09:26:45.019829 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:45.019830 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:45.019840 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:45.019903 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:45.020193 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:45.020698 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:45.021414 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:45.021601 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:45.021653 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:45.024161 (+ 2508us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.43.129:60423: BlockingRecv error: recv got EOF from 127.16.43.129:60423 (error 108)
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":75,"threads_started":1}
Jun 20 09:26:45 dist-test-slave-706q krb5kdc[24717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.43.130@KRBTEST.COM for oryx/127.16.43.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:26:45 dist-test-slave-706q krb5kdc[24717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.43.130@KRBTEST.COM for oryx/127.16.43.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:26:45.156199 25730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:45.150926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:36415 (local address 127.16.43.130:57737)
0620 09:26:45.151146 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:45.151164 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:45.151268 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:45.151618 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:45.151619 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:45.151792 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:45.151839 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:45.151841 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:45.152563 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:45.152564 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:45.153092 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:45.153094 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:45.153477 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:45.153477 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:45.153534 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:45.154023 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:26:45.154029 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:26:45.156100 (+ 2071us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.43.190:36415: Server oryx/127.16.43.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":105,"threads_started":1}
W20260620 09:26:45.156257 25729 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:26:45.151026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.130:57737 (local address 127.16.43.190:36415)
0620 09:26:45.151125 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:45.151128 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:45.151239 (+ 111us) server_negotiation.cc:407] Connection header received
0620 09:26:45.151471 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:45.151473 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:45.151488 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:45.151605 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:45.151969 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:45.152478 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:45.153190 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:45.153390 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:45.153476 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:45.156193 (+ 2717us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.43.130:57737: BlockingRecv error: recv got EOF from 127.16.43.130:57737 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Jun 20 09:26:45 dist-test-slave-706q krb5kdc[24717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.43.131@KRBTEST.COM for oryx/127.16.43.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:26:45 dist-test-slave-706q krb5kdc[24717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.43.131@KRBTEST.COM for oryx/127.16.43.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:26:45.285894 25731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:45.280932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:36415 (local address 127.16.43.131:57609)
0620 09:26:45.281097 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:45.281110 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:45.281190 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:45.281325 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:45.281326 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:45.281486 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:45.281534 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:45.281536 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:45.282180 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:45.282181 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:45.282723 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:45.282725 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:45.283075 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:45.283076 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:45.283117 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:45.283670 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:26:45.283675 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:26:45.285780 (+ 2105us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.43.190:36415: Server oryx/127.16.43.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":111,"threads_started":1}
W20260620 09:26:45.285943 25729 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:26:45.281028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.131:57609 (local address 127.16.43.190:36415)
0620 09:26:45.281120 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:45.281122 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:45.281189 (+ 67us) server_negotiation.cc:407] Connection header received
0620 09:26:45.281260 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:45.281261 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:45.281274 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:45.281319 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:45.281597 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:45.282085 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:45.282811 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:45.283002 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:45.283060 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:45.285894 (+ 2834us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.43.131:57609: BlockingRecv error: recv got EOF from 127.16.43.131:57609 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Jun 20 09:26:46 dist-test-slave-706q krb5kdc[24717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.43.129@KRBTEST.COM for oryx/127.16.43.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:26:46 dist-test-slave-706q krb5kdc[24717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.43.129@KRBTEST.COM for oryx/127.16.43.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:26:46.029559 25732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:46.024594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:36415 (local address 127.16.43.129:50013)
0620 09:26:46.024746 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:46.024762 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:46.024811 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:46.024962 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:46.024962 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:46.025174 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:46.025224 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:46.025226 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:46.025918 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:46.025919 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:46.026441 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:46.026442 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:46.026790 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:46.026791 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:46.026831 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:46.027417 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:26:46.027424 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:26:46.029500 (+ 2076us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.43.190:36415: Server oryx/127.16.43.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":101,"threads_started":1}
W20260620 09:26:46.029676 25733 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:26:46.024655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.129:50013 (local address 127.16.43.190:36415)
0620 09:26:46.024819 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:46.024821 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:46.024831 (+ 10us) server_negotiation.cc:407] Connection header received
0620 09:26:46.024857 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:46.024857 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:46.024868 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:46.024935 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:46.025312 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:46.025835 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:46.026529 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:46.026721 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:46.026779 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:46.029595 (+ 2816us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.43.129:50013: BlockingRecv error: recv got EOF from 127.16.43.129:50013 (error 108)
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":104,"threads_started":1}
Jun 20 09:26:46 dist-test-slave-706q krb5kdc[24717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.43.130@KRBTEST.COM for oryx/127.16.43.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:26:46 dist-test-slave-706q krb5kdc[24717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.43.130@KRBTEST.COM for oryx/127.16.43.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:26:46.165218 25734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:46.156748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:36415 (local address 127.16.43.130:51523)
0620 09:26:46.156955 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:46.156969 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:46.157073 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:46.157259 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:46.157260 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:46.157443 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:46.157489 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:46.157490 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:46.158164 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:46.158165 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:46.158699 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:46.158701 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:46.159099 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:46.159100 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:46.159174 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:46.163210 (+ 4036us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:26:46.163216 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:26:46.165108 (+ 1892us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.43.190:36415: Server oryx/127.16.43.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":88,"threads_started":1}
W20260620 09:26:46.165273 25733 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:26:46.156873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.130:51523 (local address 127.16.43.190:36415)
0620 09:26:46.156959 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:46.156961 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:46.157053 (+ 92us) server_negotiation.cc:407] Connection header received
0620 09:26:46.157165 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:46.157166 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:46.157185 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:46.157251 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:46.157576 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:46.158078 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:46.158789 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:46.159002 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:46.159097 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:46.165206 (+ 6109us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.43.130:51523: BlockingRecv error: recv got EOF from 127.16.43.130:51523 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Jun 20 09:26:46 dist-test-slave-706q krb5kdc[24717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.43.131@KRBTEST.COM for oryx/127.16.43.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:26:46 dist-test-slave-706q krb5kdc[24717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.43.131@KRBTEST.COM for oryx/127.16.43.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:26:46.291283 25735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:46.286479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:36415 (local address 127.16.43.131:58839)
0620 09:26:46.286667 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:46.286681 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:46.286760 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:46.286922 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:46.286923 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:46.287086 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:46.287136 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:46.287137 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:46.287935 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:46.287936 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:46.288530 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:46.288533 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:46.288866 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:46.288867 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:46.288923 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:46.289476 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:26:46.289481 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:26:46.291224 (+ 1743us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.43.190:36415: Server oryx/127.16.43.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":81,"threads_started":1}
W20260620 09:26:46.291428 25733 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:26:46.286625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.131:58839 (local address 127.16.43.190:36415)
0620 09:26:46.286699 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:46.286701 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:46.286743 (+ 42us) server_negotiation.cc:407] Connection header received
0620 09:26:46.286804 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:46.286806 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:46.286818 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:46.286868 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:46.287230 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:46.287818 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:46.288615 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:46.288794 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:46.288847 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:46.291346 (+ 2499us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.43.131:58839: BlockingRecv error: recv got EOF from 127.16.43.131:58839 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Jun 20 09:26:47 dist-test-slave-706q krb5kdc[24717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.43.129@KRBTEST.COM for oryx/127.16.43.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:26:47 dist-test-slave-706q krb5kdc[24717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.43.129@KRBTEST.COM for oryx/127.16.43.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:26:47.035017 25736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:47.030074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:36415 (local address 127.16.43.129:57391)
0620 09:26:47.030250 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:47.030261 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:47.030317 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:47.030450 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:47.030451 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:47.030651 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:47.030702 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:47.030704 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:47.031570 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:47.031572 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:47.032126 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:47.032128 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:47.032518 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:47.032519 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:47.032557 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:47.033071 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:26:47.033081 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:26:47.034895 (+ 1814us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.43.190:36415: Server oryx/127.16.43.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":117,"threads_started":1}
W20260620 09:26:47.035043 25737 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:26:47.030176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.129:57391 (local address 127.16.43.190:36415)
0620 09:26:47.030316 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:47.030317 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:47.030326 (+ 9us) server_negotiation.cc:407] Connection header received
0620 09:26:47.030367 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:47.030368 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:47.030379 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:47.030436 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:47.030843 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:47.031450 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:47.032211 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:47.032449 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:47.032509 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:47.034978 (+ 2469us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.43.129:57391: BlockingRecv error: recv got EOF from 127.16.43.129:57391 (error 108)
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":71,"threads_started":1}
Jun 20 09:26:47 dist-test-slave-706q krb5kdc[24717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.43.130@KRBTEST.COM for oryx/127.16.43.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:26:47 dist-test-slave-706q krb5kdc[24717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.43.130@KRBTEST.COM for oryx/127.16.43.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:26:47.170535 25738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:47.165784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:36415 (local address 127.16.43.130:52793)
0620 09:26:47.165980 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:47.165991 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:47.166050 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:47.166188 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:47.166189 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:47.166378 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:47.166427 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:47.166429 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:47.167093 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:47.167094 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:47.167736 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:47.167739 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:47.168094 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:47.168094 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:47.168133 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:47.168585 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:26:47.168590 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:26:47.170472 (+ 1882us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.43.190:36415: Server oryx/127.16.43.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":70,"threads_started":1}
W20260620 09:26:47.170683 25737 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:26:47.165891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.130:52793 (local address 127.16.43.190:36415)
0620 09:26:47.165962 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:47.165966 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:47.166035 (+ 69us) server_negotiation.cc:407] Connection header received
0620 09:26:47.166109 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:47.166110 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:47.166120 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:47.166162 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:47.166492 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:47.166985 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:47.167818 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:47.168025 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:47.168083 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:47.170599 (+ 2516us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.43.130:52793: BlockingRecv error: recv got EOF from 127.16.43.130:52793 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Jun 20 09:26:47 dist-test-slave-706q krb5kdc[24717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.43.131@KRBTEST.COM for oryx/127.16.43.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:26:47 dist-test-slave-706q krb5kdc[24717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.43.131@KRBTEST.COM for oryx/127.16.43.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:26:47.296813 25739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:47.291852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:36415 (local address 127.16.43.131:56617)
0620 09:26:47.292015 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:47.292027 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:47.292097 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:47.292223 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:47.292223 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:47.292383 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:47.292432 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:47.292434 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:47.293096 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:47.293097 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:47.293635 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:47.293637 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:47.293962 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:47.293963 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:47.294025 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:47.294560 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:26:47.294565 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:26:47.296712 (+ 2147us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.43.190:36415: Server oryx/127.16.43.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":88,"threads_started":1}
W20260620 09:26:47.296866 25737 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:26:47.291978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.131:56617 (local address 127.16.43.190:36415)
0620 09:26:47.292030 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:47.292033 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:47.292079 (+ 46us) server_negotiation.cc:407] Connection header received
0620 09:26:47.292155 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:47.292156 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:47.292168 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:47.292216 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:47.292503 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:47.292997 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:47.293718 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:47.293894 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:47.293944 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:47.296802 (+ 2858us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.43.131:56617: BlockingRecv error: recv got EOF from 127.16.43.131:56617 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
Jun 20 09:26:48 dist-test-slave-706q krb5kdc[24717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.43.129@KRBTEST.COM for oryx/127.16.43.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:26:48 dist-test-slave-706q krb5kdc[24717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.43.129@KRBTEST.COM for oryx/127.16.43.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:26:48.040098 25740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:48.035608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:36415 (local address 127.16.43.129:47693)
0620 09:26:48.035756 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:48.035775 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:48.035823 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:48.035983 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:48.035984 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:48.036149 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:48.036222 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:48.036223 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:48.036865 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:48.036866 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:48.037407 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:48.037408 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:48.037750 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:48.037751 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:48.037799 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:48.038232 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:26:48.038236 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:26:48.040037 (+ 1801us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.43.190:36415: Server oryx/127.16.43.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":83,"threads_started":1}
W20260620 09:26:48.040246 25741 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:26:48.035659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.129:47693 (local address 127.16.43.190:36415)
0620 09:26:48.035816 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:48.035817 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:48.035825 (+ 8us) server_negotiation.cc:407] Connection header received
0620 09:26:48.035878 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:48.035878 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:48.035890 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:48.035941 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:48.036295 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:48.036783 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:48.037480 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:48.037687 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:48.037737 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:48.040148 (+ 2411us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.43.129:47693: BlockingRecv error: recv got EOF from 127.16.43.129:47693 (error 108)
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":85,"threads_started":1}
Jun 20 09:26:48 dist-test-slave-706q krb5kdc[24717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.43.130@KRBTEST.COM for oryx/127.16.43.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:26:48 dist-test-slave-706q krb5kdc[24717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.43.130@KRBTEST.COM for oryx/127.16.43.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:26:48.176281 25742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:48.171097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:36415 (local address 127.16.43.130:33615)
0620 09:26:48.171226 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:48.171238 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:48.171297 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:48.171466 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:48.171467 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:48.171644 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:48.171693 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:48.171695 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:48.172325 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:48.172326 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:48.172886 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:48.172888 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:48.173222 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:48.173223 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:48.173264 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:48.173722 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:26:48.173726 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:26:48.176175 (+ 2449us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.43.190:36415: Server oryx/127.16.43.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":78,"threads_started":1}
W20260620 09:26:48.176342 25741 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:26:48.171204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.130:33615 (local address 127.16.43.190:36415)
0620 09:26:48.171268 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:48.171270 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:48.171286 (+ 16us) server_negotiation.cc:407] Connection header received
0620 09:26:48.171377 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:48.171378 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:48.171388 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:48.171448 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:48.171753 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:48.172248 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:48.172971 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:48.173156 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:48.173211 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:48.176272 (+ 3061us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.43.130:33615: BlockingRecv error: recv got EOF from 127.16.43.130:33615 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Jun 20 09:26:48 dist-test-slave-706q krb5kdc[24717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.43.131@KRBTEST.COM for oryx/127.16.43.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:26:48 dist-test-slave-706q krb5kdc[24717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.43.131@KRBTEST.COM for oryx/127.16.43.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:26:48.302124 25743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:48.297362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:36415 (local address 127.16.43.131:59883)
0620 09:26:48.297528 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:48.297551 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:48.297614 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:48.297826 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:48.297827 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:48.297988 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:48.298038 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:48.298039 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:48.298780 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:48.298780 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:48.299329 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:48.299331 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:48.299826 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:48.299827 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:48.299873 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:48.300325 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:26:48.300330 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:26:48.302063 (+ 1733us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.43.190:36415: Server oryx/127.16.43.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":101,"threads_started":1}
W20260620 09:26:48.302399 25741 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:26:48.297534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.131:59883 (local address 127.16.43.190:36415)
0620 09:26:48.297583 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:48.297585 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:48.297602 (+ 17us) server_negotiation.cc:407] Connection header received
0620 09:26:48.297738 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:48.297738 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:48.297751 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:48.297806 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:48.298111 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:48.298706 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:48.299437 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:48.299732 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:48.299799 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:48.302327 (+ 2528us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.43.131:59883: BlockingRecv error: recv got EOF from 127.16.43.131:59883 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Jun 20 09:26:49 dist-test-slave-706q krb5kdc[24717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.43.129@KRBTEST.COM for oryx/127.16.43.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:26:49 dist-test-slave-706q krb5kdc[24717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.43.129@KRBTEST.COM for oryx/127.16.43.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:26:49.045234 25744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:49.040629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:36415 (local address 127.16.43.129:45235)
0620 09:26:49.040772 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:49.040785 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:49.040865 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:49.041020 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:49.041021 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:49.041179 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:49.041226 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:49.041228 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:49.041921 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:49.041922 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:49.042450 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:49.042452 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:49.042821 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:49.042822 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:49.042868 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:49.043314 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:26:49.043319 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:26:49.045176 (+ 1857us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.43.190:36415: Server oryx/127.16.43.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":64,"threads_started":1}
W20260620 09:26:49.045390 25745 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:26:49.040731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.129:45235 (local address 127.16.43.190:36415)
0620 09:26:49.040881 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:49.040883 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:49.040895 (+ 12us) server_negotiation.cc:407] Connection header received
0620 09:26:49.040922 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:49.040922 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:49.040935 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:49.041009 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:49.041301 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:49.041838 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:49.042548 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:49.042747 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:49.042804 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:49.045307 (+ 2503us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.43.129:45235: BlockingRecv error: recv got EOF from 127.16.43.129:45235 (error 108)
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":70,"threads_started":1}
Jun 20 09:26:49 dist-test-slave-706q krb5kdc[24717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.43.130@KRBTEST.COM for oryx/127.16.43.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:26:49 dist-test-slave-706q krb5kdc[24717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.43.130@KRBTEST.COM for oryx/127.16.43.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:26:49.181663 25746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:49.176849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:36415 (local address 127.16.43.130:55697)
0620 09:26:49.176990 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:49.177008 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:49.177063 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:49.177250 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:49.177251 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:49.177408 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:49.177455 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:49.177457 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:49.178099 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:49.178099 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:49.178629 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:49.178631 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:49.178956 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:49.178956 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:49.178995 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:49.179484 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:26:49.179489 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:26:49.181590 (+ 2101us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.43.190:36415: Server oryx/127.16.43.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":88,"threads_started":1}
W20260620 09:26:49.181718 25745 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:26:49.176922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.130:55697 (local address 127.16.43.190:36415)
0620 09:26:49.177114 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:49.177116 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:49.177134 (+ 18us) server_negotiation.cc:407] Connection header received
0620 09:26:49.177162 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:49.177163 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:49.177173 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:49.177232 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:49.177524 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:49.178032 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:49.178706 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:49.178890 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:49.178941 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:49.181671 (+ 2730us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.43.130:55697: BlockingRecv error: recv got EOF from 127.16.43.130:55697 (error 108)
Metrics: {"server-negotiator.queue_time_us":166}
Jun 20 09:26:49 dist-test-slave-706q krb5kdc[24717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.43.131@KRBTEST.COM for oryx/127.16.43.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:26:49 dist-test-slave-706q krb5kdc[24717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.43.131@KRBTEST.COM for oryx/127.16.43.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:26:49.307446 25747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:49.302734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:36415 (local address 127.16.43.131:42893)
0620 09:26:49.302925 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:49.302937 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:49.303012 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:49.303159 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:49.303160 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:49.303317 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:49.303366 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:49.303368 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:49.304108 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:49.304108 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:49.304638 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:49.304640 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:49.304955 (+ 315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:49.304956 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:49.304996 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:49.305517 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:26:49.305522 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:26:49.307387 (+ 1865us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.43.190:36415: Server oryx/127.16.43.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":86,"threads_started":1}
W20260620 09:26:49.307579 25745 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:26:49.302876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.131:42893 (local address 127.16.43.190:36415)
0620 09:26:49.302944 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:49.302946 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:49.303006 (+ 60us) server_negotiation.cc:407] Connection header received
0620 09:26:49.303054 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:49.303055 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:49.303071 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:49.303140 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:49.303427 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:49.304031 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:49.304709 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:49.304887 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:49.304942 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:49.307480 (+ 2538us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.43.131:42893: BlockingRecv error: recv got EOF from 127.16.43.131:42893 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Jun 20 09:26:50 dist-test-slave-706q krb5kdc[24717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.43.129@KRBTEST.COM for oryx/127.16.43.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:26:50 dist-test-slave-706q krb5kdc[24717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.43.129@KRBTEST.COM for oryx/127.16.43.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:26:50.051162 25748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:50.045785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:36415 (local address 127.16.43.129:53561)
0620 09:26:50.045968 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:50.045980 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:50.046042 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:50.046281 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:50.046282 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:50.046445 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:50.046497 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:50.046499 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:50.047318 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:50.047319 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:50.047944 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:50.047946 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:50.048479 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:50.048480 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:50.048548 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:50.049232 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:26:50.049238 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:26:50.051102 (+ 1864us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.43.190:36415: Server oryx/127.16.43.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":74,"threads_started":1}
W20260620 09:26:50.051318 25749 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:26:50.046012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.129:53561 (local address 127.16.43.190:36415)
0620 09:26:50.046146 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:50.046148 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:50.046157 (+ 9us) server_negotiation.cc:407] Connection header received
0620 09:26:50.046182 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:50.046183 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:50.046194 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:50.046261 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:50.046618 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:50.047221 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:50.048052 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:50.048353 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:50.048447 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:50.051218 (+ 2771us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.43.129:53561: BlockingRecv error: recv got EOF from 127.16.43.129:53561 (error 108)
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":82,"threads_started":1}
Jun 20 09:26:50 dist-test-slave-706q krb5kdc[24717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.43.130@KRBTEST.COM for oryx/127.16.43.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:26:50 dist-test-slave-706q krb5kdc[24717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.43.130@KRBTEST.COM for oryx/127.16.43.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:26:50.187148 25750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:50.182282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:36415 (local address 127.16.43.130:40403)
0620 09:26:50.182503 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:50.182516 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:50.182590 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:50.182798 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:50.182799 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:50.183062 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:50.183126 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:50.183128 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:50.183979 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:50.183980 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:50.184510 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:50.184512 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:50.184832 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:50.184833 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:50.184871 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:50.185316 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:26:50.185320 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:26:50.187089 (+ 1769us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.43.190:36415: Server oryx/127.16.43.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":120,"threads_started":1}
W20260620 09:26:50.187253 25749 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:26:50.182432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.130:40403 (local address 127.16.43.190:36415)
0620 09:26:50.182503 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:50.182505 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:50.182599 (+ 94us) server_negotiation.cc:407] Connection header received
0620 09:26:50.182683 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:50.182683 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:50.182697 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:50.182764 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:50.183224 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:50.183909 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:50.184589 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:50.184769 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:50.184820 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:50.187160 (+ 2340us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.43.130:40403: BlockingRecv error: recv got EOF from 127.16.43.130:40403 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Jun 20 09:26:50 dist-test-slave-706q krb5kdc[24717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.43.131@KRBTEST.COM for oryx/127.16.43.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:26:50 dist-test-slave-706q krb5kdc[24717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.43.131@KRBTEST.COM for oryx/127.16.43.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:26:50.312892 25751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:50.308015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:36415 (local address 127.16.43.131:35435)
0620 09:26:50.308205 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:50.308222 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:50.308280 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:50.308441 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:50.308441 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:50.308643 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:50.308691 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:50.308692 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:50.309379 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:50.309380 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:50.309900 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:50.309901 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:50.310275 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:50.310275 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:50.310315 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:50.310761 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:26:50.310766 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:26:50.312822 (+ 2056us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.43.190:36415: Server oryx/127.16.43.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":140,"threads_started":1}
W20260620 09:26:50.312959 25749 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:26:50.308148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.131:35435 (local address 127.16.43.190:36415)
0620 09:26:50.308220 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:50.308222 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:50.308267 (+ 45us) server_negotiation.cc:407] Connection header received
0620 09:26:50.308331 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:50.308332 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:50.308344 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:50.308440 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:50.308788 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:50.309300 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:50.309986 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:50.310204 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:50.310261 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:50.312912 (+ 2651us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.43.131:35435: BlockingRecv error: recv got EOF from 127.16.43.131:35435 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Jun 20 09:26:51 dist-test-slave-706q krb5kdc[24717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.43.129@KRBTEST.COM for oryx/127.16.43.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:26:51 dist-test-slave-706q krb5kdc[24717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.43.129@KRBTEST.COM for oryx/127.16.43.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:26:51.059226 25752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:51.051698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:36415 (local address 127.16.43.129:42629)
0620 09:26:51.051866 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:51.051876 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:51.051929 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:51.052070 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:51.052071 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:51.052242 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:51.052290 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:51.052292 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:51.052958 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:51.052958 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:51.053467 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:51.053468 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:51.053800 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:51.053800 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:51.053838 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:51.057066 (+ 3228us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:26:51.057072 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:26:51.059166 (+ 2094us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.43.190:36415: Server oryx/127.16.43.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":107,"threads_started":1}
W20260620 09:26:51.059345 25753 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:26:51.051820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.129:42629 (local address 127.16.43.190:36415)
0620 09:26:51.051926 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:51.051928 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:51.051939 (+ 11us) server_negotiation.cc:407] Connection header received
0620 09:26:51.051972 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:51.051973 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:51.051987 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:51.052053 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:51.052366 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:51.052880 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:51.053540 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:51.053733 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:51.053789 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:51.059263 (+ 5474us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.43.129:42629: BlockingRecv error: recv got EOF from 127.16.43.129:42629 (error 108)
Metrics: {"server-negotiator.queue_time_us":82,"thread_start_us":50,"threads_started":1}
Jun 20 09:26:51 dist-test-slave-706q krb5kdc[24717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.43.130@KRBTEST.COM for oryx/127.16.43.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:26:51 dist-test-slave-706q krb5kdc[24717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.43.130@KRBTEST.COM for oryx/127.16.43.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:26:51.195242 25754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:51.187736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:36415 (local address 127.16.43.130:50801)
0620 09:26:51.187912 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:51.187923 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:51.187989 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:51.188143 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:51.188144 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:51.188307 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:51.188352 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:51.188354 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:51.188981 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:51.188982 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:51.189509 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:51.189511 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:51.190019 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:51.190021 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:51.190124 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:51.193383 (+ 3259us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:26:51.193395 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:26:51.195183 (+ 1788us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.43.190:36415: Server oryx/127.16.43.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":91,"threads_started":1}
W20260620 09:26:51.195411 25753 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:26:51.187863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.130:50801 (local address 127.16.43.190:36415)
0620 09:26:51.187915 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:51.187917 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:51.187971 (+ 54us) server_negotiation.cc:407] Connection header received
0620 09:26:51.188036 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:51.188036 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:51.188047 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:51.188106 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:51.188416 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:51.188917 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:51.189615 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:51.189853 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:51.189918 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:51.195309 (+ 5391us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.43.130:50801: BlockingRecv error: recv got EOF from 127.16.43.130:50801 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Jun 20 09:26:51 dist-test-slave-706q krb5kdc[24717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.43.131@KRBTEST.COM for oryx/127.16.43.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:26:51 dist-test-slave-706q krb5kdc[24717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.43.131@KRBTEST.COM for oryx/127.16.43.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:26:51.318141 25755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:51.313483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:36415 (local address 127.16.43.131:36233)
0620 09:26:51.313665 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:51.313675 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:51.313755 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:51.313921 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:51.313922 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:51.314117 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:51.314163 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:51.314166 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:51.314843 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:51.314844 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:51.315374 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:51.315375 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:51.315786 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:51.315786 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:51.315848 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:51.316267 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:26:51.316272 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:26:51.318081 (+ 1809us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.43.190:36415: Server oryx/127.16.43.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":84,"threads_started":1}
W20260620 09:26:51.318286 25753 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:26:51.313552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.131:36233 (local address 127.16.43.190:36415)
0620 09:26:51.313628 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:51.313631 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:51.313742 (+ 111us) server_negotiation.cc:407] Connection header received
0620 09:26:51.313829 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:51.313830 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:51.313850 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:51.313914 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:51.314248 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:51.314757 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:51.315458 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:51.315712 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:51.315787 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:51.318207 (+ 2420us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.43.131:36233: BlockingRecv error: recv got EOF from 127.16.43.131:36233 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Jun 20 09:26:52 dist-test-slave-706q krb5kdc[24717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.43.129@KRBTEST.COM for oryx/127.16.43.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:26:52 dist-test-slave-706q krb5kdc[24717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.43.129@KRBTEST.COM for oryx/127.16.43.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:26:52.064833 25756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:52.059745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:36415 (local address 127.16.43.129:40787)
0620 09:26:52.059898 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:52.059908 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:52.059964 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:52.060118 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:52.060119 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:52.060281 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:52.060332 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:52.060333 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:52.061035 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:52.061036 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:52.061652 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:52.061654 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:52.062111 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:52.062112 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:52.062157 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:52.062729 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:26:52.062735 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:26:52.064723 (+ 1988us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.43.190:36415: Server oryx/127.16.43.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":91,"threads_started":1}
W20260620 09:26:52.064877 25757 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:26:52.059788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.129:40787 (local address 127.16.43.190:36415)
0620 09:26:52.059913 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:52.059915 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:52.059961 (+ 46us) server_negotiation.cc:407] Connection header received
0620 09:26:52.060004 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:52.060005 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:52.060015 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:52.060098 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:52.060394 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:52.060968 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:52.061756 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:52.062032 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:52.062099 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:52.064805 (+ 2706us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.43.129:40787: BlockingRecv error: recv got EOF from 127.16.43.129:40787 (error 108)
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":76,"threads_started":1}
Jun 20 09:26:52 dist-test-slave-706q krb5kdc[24717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.43.130@KRBTEST.COM for oryx/127.16.43.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:26:52 dist-test-slave-706q krb5kdc[24717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.43.130@KRBTEST.COM for oryx/127.16.43.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:26:52.200804 25758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:52.195830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:36415 (local address 127.16.43.130:48575)
0620 09:26:52.195989 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:52.196001 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:52.196064 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:52.196210 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:52.196211 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:52.196372 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:52.196418 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:52.196420 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:52.197133 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:52.197134 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:52.197710 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:52.197712 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:52.198186 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:52.198188 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:52.198258 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:52.198770 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:26:52.198775 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:26:52.200706 (+ 1931us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.43.190:36415: Server oryx/127.16.43.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":106,"threads_started":1}
W20260620 09:26:52.200853 25757 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:26:52.195888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.130:48575 (local address 127.16.43.190:36415)
0620 09:26:52.195975 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:52.195977 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:52.196053 (+ 76us) server_negotiation.cc:407] Connection header received
0620 09:26:52.196123 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:52.196124 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:52.196135 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:52.196192 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:52.196484 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:52.197024 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:52.197849 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:52.198070 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:52.198134 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:52.200787 (+ 2653us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.43.130:48575: BlockingRecv error: recv got EOF from 127.16.43.130:48575 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Jun 20 09:26:52 dist-test-slave-706q krb5kdc[24717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.43.131@KRBTEST.COM for oryx/127.16.43.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:26:52 dist-test-slave-706q krb5kdc[24717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.43.131@KRBTEST.COM for oryx/127.16.43.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:26:52.323478 25759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:52.318677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:36415 (local address 127.16.43.131:39593)
0620 09:26:52.318842 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:52.318857 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:52.318925 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:52.319099 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:52.319100 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:52.319260 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:52.319309 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:52.319310 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:52.320001 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:52.320002 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:52.320645 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:52.320647 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:52.321041 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:52.321042 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:52.321108 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:52.321556 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:26:52.321560 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:26:52.323372 (+ 1812us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.43.190:36415: Server oryx/127.16.43.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":60,"threads_started":1}
W20260620 09:26:52.323514 25757 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:26:52.318858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.131:39593 (local address 127.16.43.190:36415)
0620 09:26:52.318933 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:52.318935 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:52.318968 (+ 33us) server_negotiation.cc:407] Connection header received
0620 09:26:52.319001 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:52.319001 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:52.319013 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:52.319054 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:52.319396 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:52.319930 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:52.320725 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:52.320959 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:52.321022 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:52.323449 (+ 2427us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.43.131:39593: BlockingRecv error: recv got EOF from 127.16.43.131:39593 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Jun 20 09:26:53 dist-test-slave-706q krb5kdc[24717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.43.129@KRBTEST.COM for oryx/127.16.43.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:26:53 dist-test-slave-706q krb5kdc[24717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.43.129@KRBTEST.COM for oryx/127.16.43.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:26:53.070531 25760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:53.065391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:36415 (local address 127.16.43.129:36811)
0620 09:26:53.065541 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:53.065552 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:53.065610 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:53.065805 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:53.065805 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:53.065973 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:53.066023 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:53.066025 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:53.066728 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:53.066729 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:53.067346 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:53.067349 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:53.067788 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:53.067789 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:53.067842 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:53.068332 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:26:53.068337 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:26:53.070434 (+ 2097us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.43.190:36415: Server oryx/127.16.43.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":88,"threads_started":1}
W20260620 09:26:53.070600 25761 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:26:53.065497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.129:36811 (local address 127.16.43.190:36415)
0620 09:26:53.065660 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:53.065662 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:53.065671 (+ 9us) server_negotiation.cc:407] Connection header received
0620 09:26:53.065697 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:53.065698 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:53.065709 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:53.065780 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:53.066099 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:53.066616 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:53.067467 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:53.067697 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:53.067761 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:53.070530 (+ 2769us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.43.129:36811: BlockingRecv error: recv got EOF from 127.16.43.129:36811 (error 108)
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":74,"threads_started":1}
Jun 20 09:26:53 dist-test-slave-706q krb5kdc[24717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.43.130@KRBTEST.COM for oryx/127.16.43.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:26:53 dist-test-slave-706q krb5kdc[24717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.43.130@KRBTEST.COM for oryx/127.16.43.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:26:53.206655 25762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:53.201293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:36415 (local address 127.16.43.130:35115)
0620 09:26:53.201425 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:53.201435 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:53.201494 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:53.201639 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:53.201639 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:53.201861 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:53.201919 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:53.201922 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:53.202882 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:53.202883 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:53.203499 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:53.203502 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:53.203983 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:53.203984 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:53.204036 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:53.204637 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:26:53.204645 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:26:53.206582 (+ 1937us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.43.190:36415: Server oryx/127.16.43.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":68,"threads_started":1}
W20260620 09:26:53.206779 25761 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:26:53.201361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.130:35115 (local address 127.16.43.190:36415)
0620 09:26:53.201433 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:53.201435 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:53.201492 (+ 57us) server_negotiation.cc:407] Connection header received
0620 09:26:53.201545 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:53.201546 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:53.201557 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:53.201633 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:53.202021 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:53.202763 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:53.203622 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:53.203834 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:53.203893 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:53.206681 (+ 2788us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.43.130:35115: BlockingRecv error: recv got EOF from 127.16.43.130:35115 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Jun 20 09:26:53 dist-test-slave-706q krb5kdc[24717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.43.131@KRBTEST.COM for oryx/127.16.43.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:26:53 dist-test-slave-706q krb5kdc[24717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.43.131@KRBTEST.COM for oryx/127.16.43.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:26:53.328588 25763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:53.324043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:36415 (local address 127.16.43.131:46253)
0620 09:26:53.324186 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:53.324199 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:53.324265 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:53.324417 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:53.324418 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:53.324591 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:53.324638 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:53.324639 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:53.325284 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:53.325285 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:53.325854 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:53.325855 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:53.326265 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:53.326266 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:53.326317 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:53.326767 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:26:53.326771 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:26:53.328520 (+ 1749us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.43.190:36415: Server oryx/127.16.43.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":94,"threads_started":1}
W20260620 09:26:53.328725 25761 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:26:53.324107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.131:46253 (local address 127.16.43.190:36415)
0620 09:26:53.324182 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:53.324184 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:53.324299 (+ 115us) server_negotiation.cc:407] Connection header received
0620 09:26:53.324327 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:53.324328 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:53.324342 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:53.324401 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:53.324709 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:53.325211 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:53.325933 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:53.326178 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:53.326236 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:53.328622 (+ 2386us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.43.131:46253: BlockingRecv error: recv got EOF from 127.16.43.131:46253 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Jun 20 09:26:54 dist-test-slave-706q krb5kdc[24717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.43.129@KRBTEST.COM for oryx/127.16.43.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:26:54 dist-test-slave-706q krb5kdc[24717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.43.129@KRBTEST.COM for oryx/127.16.43.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:26:54.076180 25764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:54.071217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:36415 (local address 127.16.43.129:53225)
0620 09:26:54.071358 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:54.071369 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:54.071431 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:54.071602 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:54.071603 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:54.071781 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:54.071832 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:54.071834 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:54.072512 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:54.072513 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:54.073044 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:54.073045 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:54.073465 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:54.073466 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:54.073525 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:54.074004 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:26:54.074008 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:26:54.076121 (+ 2113us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.43.190:36415: Server oryx/127.16.43.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":88,"threads_started":1}
W20260620 09:26:54.076361 25765 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:26:54.071321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.129:53225 (local address 127.16.43.190:36415)
0620 09:26:54.071463 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:54.071464 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:54.071473 (+ 9us) server_negotiation.cc:407] Connection header received
0620 09:26:54.071498 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:54.071498 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:54.071508 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:54.071595 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:54.071901 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:54.072438 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:54.073153 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:54.073356 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:54.073465 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:54.076240 (+ 2775us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.43.129:53225: BlockingRecv error: recv got EOF from 127.16.43.129:53225 (error 108)
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":83,"threads_started":1}
Jun 20 09:26:54 dist-test-slave-706q krb5kdc[24717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.43.130@KRBTEST.COM for oryx/127.16.43.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:26:54 dist-test-slave-706q krb5kdc[24717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.43.130@KRBTEST.COM for oryx/127.16.43.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:26:54.212183 25766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:54.207149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:36415 (local address 127.16.43.130:59195)
0620 09:26:54.207335 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:54.207366 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:54.207457 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:54.207620 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:54.207621 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:54.207809 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:54.207858 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:54.207860 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:54.208555 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:54.208556 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:54.209162 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:54.209164 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:54.209509 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:54.209509 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:54.209549 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:54.210015 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:26:54.210020 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:26:54.212084 (+ 2064us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.43.190:36415: Server oryx/127.16.43.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":72,"threads_started":1}
W20260620 09:26:54.212247 25765 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:26:54.207309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.130:59195 (local address 127.16.43.190:36415)
0620 09:26:54.207382 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:54.207384 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:54.207436 (+ 52us) server_negotiation.cc:407] Connection header received
0620 09:26:54.207508 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:54.207508 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:54.207519 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:54.207590 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:54.207926 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:54.208451 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:54.209249 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:54.209439 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:54.209496 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:54.212185 (+ 2689us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.43.130:59195: BlockingRecv error: recv got EOF from 127.16.43.130:59195 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Jun 20 09:26:54 dist-test-slave-706q krb5kdc[24717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.43.131@KRBTEST.COM for oryx/127.16.43.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:26:54 dist-test-slave-706q krb5kdc[24717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.43.131@KRBTEST.COM for oryx/127.16.43.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:26:54.334342 25767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:54.329233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:36415 (local address 127.16.43.131:39125)
0620 09:26:54.329441 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:54.329459 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:54.329531 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:54.329683 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:54.329684 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:54.329863 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:54.329915 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:54.329916 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:54.330694 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:54.330696 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:54.331276 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:54.331280 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:54.331735 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:54.331736 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:54.331784 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:54.332289 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:26:54.332293 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:26:54.334244 (+ 1951us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.43.190:36415: Server oryx/127.16.43.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":91,"threads_started":1}
W20260620 09:26:54.334403 25765 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:26:54.329373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.131:39125 (local address 127.16.43.190:36415)
0620 09:26:54.329442 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:54.329444 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:54.329521 (+ 77us) server_negotiation.cc:407] Connection header received
0620 09:26:54.329590 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:54.329591 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:54.329604 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:54.329655 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:54.329981 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:54.330617 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:54.331371 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:54.331665 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:54.331718 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:54.334322 (+ 2604us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.43.131:39125: BlockingRecv error: recv got EOF from 127.16.43.131:39125 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Jun 20 09:26:55 dist-test-slave-706q krb5kdc[24717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.43.129@KRBTEST.COM for oryx/127.16.43.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:26:55 dist-test-slave-706q krb5kdc[24717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.43.129@KRBTEST.COM for oryx/127.16.43.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:26:55.081326 25768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:55.076782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:36415 (local address 127.16.43.129:47657)
0620 09:26:55.077023 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:55.077035 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:55.077114 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:55.077280 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:55.077281 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:55.077459 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:55.077506 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:55.077508 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:55.078185 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:55.078186 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:55.078700 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:55.078702 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:55.079020 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:55.079021 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:55.079059 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:55.079501 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:26:55.079505 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:26:55.081266 (+ 1761us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.43.190:36415: Server oryx/127.16.43.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":103,"threads_started":1}
W20260620 09:26:55.081451 25769 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:26:55.076921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.129:47657 (local address 127.16.43.190:36415)
0620 09:26:55.077060 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:55.077063 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:55.077097 (+ 34us) server_negotiation.cc:407] Connection header received
0620 09:26:55.077166 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:55.077167 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:55.077178 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:55.077238 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:55.077581 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:55.078108 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:55.078771 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:55.078956 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:55.079006 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:55.081365 (+ 2359us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.43.129:47657: BlockingRecv error: recv got EOF from 127.16.43.129:47657 (error 108)
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":90,"threads_started":1}
Jun 20 09:26:55 dist-test-slave-706q krb5kdc[24717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.43.130@KRBTEST.COM for oryx/127.16.43.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:26:55 dist-test-slave-706q krb5kdc[24717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.43.130@KRBTEST.COM for oryx/127.16.43.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:26:55.218035 25770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:55.212864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:36415 (local address 127.16.43.130:36529)
0620 09:26:55.213028 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:55.213040 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:55.213167 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:55.213343 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:55.213344 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:55.213530 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:55.213574 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:55.213575 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:55.214261 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:55.214262 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:55.214798 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:55.214799 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:55.215174 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:55.215175 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:55.215232 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:55.215760 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:26:55.215765 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:26:55.217976 (+ 2211us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.43.190:36415: Server oryx/127.16.43.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":74,"threads_started":1}
W20260620 09:26:55.218174 25769 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:26:55.213102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.130:36529 (local address 127.16.43.190:36415)
0620 09:26:55.213183 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:55.213185 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:55.213214 (+ 29us) server_negotiation.cc:407] Connection header received
0620 09:26:55.213244 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:55.213247 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:55.213263 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:55.213336 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:55.213680 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:55.214174 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:55.214890 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:55.215086 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:55.215175 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:55.218091 (+ 2916us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.43.130:36529: BlockingRecv error: recv got EOF from 127.16.43.130:36529 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Jun 20 09:26:55 dist-test-slave-706q krb5kdc[24717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.43.131@KRBTEST.COM for oryx/127.16.43.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:26:55 dist-test-slave-706q krb5kdc[24717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.43.131@KRBTEST.COM for oryx/127.16.43.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:26:55.340086 25771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:55.334902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:36415 (local address 127.16.43.131:54231)
0620 09:26:55.335080 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:55.335098 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:55.335155 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:55.335340 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:55.335341 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:55.335656 (+ 315us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:55.335715 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:55.335717 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:55.336503 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:55.336504 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:55.337091 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:55.337094 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:55.337439 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:55.337440 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:55.337488 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:55.337985 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:26:55.337991 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:26:55.340017 (+ 2026us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.43.190:36415: Server oryx/127.16.43.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":119,"threads_started":1}
W20260620 09:26:55.340229 25769 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:26:55.335054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.131:54231 (local address 127.16.43.190:36415)
0620 09:26:55.335131 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:55.335133 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:55.335143 (+ 10us) server_negotiation.cc:407] Connection header received
0620 09:26:55.335219 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:55.335220 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:55.335232 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:55.335294 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:55.335818 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:55.336401 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:55.337174 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:55.337353 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:55.337406 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:55.340147 (+ 2741us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.43.131:54231: BlockingRecv error: recv got EOF from 127.16.43.131:54231 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Jun 20 09:26:56 dist-test-slave-706q krb5kdc[24717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.43.129@KRBTEST.COM for oryx/127.16.43.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:26:56 dist-test-slave-706q krb5kdc[24717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.43.129@KRBTEST.COM for oryx/127.16.43.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:26:56.086400 25772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:56.081839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:36415 (local address 127.16.43.129:54223)
0620 09:26:56.081982 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:56.081993 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:56.082049 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:56.082208 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:56.082209 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:56.082372 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:56.082421 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:56.082423 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:56.083048 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:56.083049 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:56.083681 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:56.083684 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:56.084022 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:56.084023 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:56.084068 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:56.084522 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:26:56.084526 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:26:56.086339 (+ 1813us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.43.190:36415: Server oryx/127.16.43.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":90,"threads_started":1}
W20260620 09:26:56.086552 25773 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:26:56.081886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.129:54223 (local address 127.16.43.190:36415)
0620 09:26:56.082041 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:56.082043 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:56.082051 (+ 8us) server_negotiation.cc:407] Connection header received
0620 09:26:56.082111 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:56.082112 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:56.082122 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:56.082188 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:56.082488 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:56.082981 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:56.083761 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:56.083953 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:56.084003 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:56.086454 (+ 2451us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.43.129:54223: BlockingRecv error: recv got EOF from 127.16.43.129:54223 (error 108)
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":94,"threads_started":1}
Jun 20 09:26:56 dist-test-slave-706q krb5kdc[24717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.43.130@KRBTEST.COM for oryx/127.16.43.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:26:56 dist-test-slave-706q krb5kdc[24717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.43.130@KRBTEST.COM for oryx/127.16.43.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:26:56.226151 25774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:56.218565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:36415 (local address 127.16.43.130:50555)
0620 09:26:56.218730 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:56.218741 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:56.218799 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:56.218935 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:56.218936 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:56.219101 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:56.219149 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:56.219151 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:56.219836 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:56.219836 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:56.220360 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:56.220361 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:56.220691 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:56.220691 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:56.220731 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:56.224176 (+ 3445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:26:56.224182 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:26:56.226089 (+ 1907us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.43.190:36415: Server oryx/127.16.43.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":102,"threads_started":1}
W20260620 09:26:56.226276 25773 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:26:56.218621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.130:50555 (local address 127.16.43.190:36415)
0620 09:26:56.218716 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:56.218718 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:56.218802 (+ 84us) server_negotiation.cc:407] Connection header received
0620 09:26:56.218859 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:56.218860 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:56.218870 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:56.218916 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:56.219214 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:56.219768 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:56.220437 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:56.220624 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:56.220677 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:56.226203 (+ 5526us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.43.130:50555: BlockingRecv error: recv got EOF from 127.16.43.130:50555 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Jun 20 09:26:56 dist-test-slave-706q krb5kdc[24717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.43.131@KRBTEST.COM for oryx/127.16.43.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:26:56 dist-test-slave-706q krb5kdc[24717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.43.131@KRBTEST.COM for oryx/127.16.43.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:26:56.345724 25775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:56.340644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:36415 (local address 127.16.43.131:60845)
0620 09:26:56.340852 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:56.340862 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:56.340940 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:56.341150 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:56.341151 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:56.341349 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:56.341395 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:56.341397 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:56.342239 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:56.342240 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:56.342901 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:56.342903 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:56.343298 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:56.343299 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:56.343339 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:56.343823 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:26:56.343827 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:26:56.345656 (+ 1829us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.43.190:36415: Server oryx/127.16.43.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":108,"threads_started":1}
W20260620 09:26:56.345840 25773 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:26:56.340715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.131:60845 (local address 127.16.43.190:36415)
0620 09:26:56.340788 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:56.340790 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:56.340926 (+ 136us) server_negotiation.cc:407] Connection header received
0620 09:26:56.340999 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:56.341000 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:56.341015 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:56.341138 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:56.341490 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:56.342145 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:56.343012 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:56.343217 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:56.343283 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:56.345762 (+ 2479us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.43.131:60845: BlockingRecv error: recv got EOF from 127.16.43.131:60845 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Jun 20 09:26:57 dist-test-slave-706q krb5kdc[24717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.43.129@KRBTEST.COM for oryx/127.16.43.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:26:57 dist-test-slave-706q krb5kdc[24717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.43.129@KRBTEST.COM for oryx/127.16.43.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:26:57.092090 25776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:57.087048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:36415 (local address 127.16.43.129:56937)
0620 09:26:57.087233 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:57.087243 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:57.087296 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:57.087573 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:57.087574 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:57.087795 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:57.087883 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:57.087886 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:57.088770 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:57.088771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:57.089309 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:57.089311 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:57.089692 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:57.089693 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:57.089731 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:57.090153 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:26:57.090158 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:26:57.092032 (+ 1874us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.43.190:36415: Server oryx/127.16.43.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":97,"threads_started":1}
W20260620 09:26:57.092221 25777 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:26:57.087207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.129:56937 (local address 127.16.43.190:36415)
0620 09:26:57.087365 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:57.087367 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:57.087376 (+ 9us) server_negotiation.cc:407] Connection header received
0620 09:26:57.087400 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:57.087401 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:57.087411 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:57.087549 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:57.087976 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:57.088674 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:57.089409 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:57.089604 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:57.089683 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:57.092127 (+ 2444us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.43.129:56937: BlockingRecv error: recv got EOF from 127.16.43.129:56937 (error 108)
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":94,"threads_started":1}
Jun 20 09:26:57 dist-test-slave-706q krb5kdc[24717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.43.130@KRBTEST.COM for oryx/127.16.43.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:26:57 dist-test-slave-706q krb5kdc[24717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.43.130@KRBTEST.COM for oryx/127.16.43.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:26:57.232177 25778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:57.226756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:36415 (local address 127.16.43.130:38049)
0620 09:26:57.226938 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:57.226954 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:57.227034 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:57.227228 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:57.227229 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:57.227413 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:57.227460 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:57.227462 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:57.228169 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:57.228170 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:57.228709 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:57.228711 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:57.229088 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:57.229089 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:57.229147 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:57.229601 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:26:57.229605 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:26:57.232080 (+ 2475us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.43.190:36415: Server oryx/127.16.43.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":84,"threads_started":1}
W20260620 09:26:57.232220 25777 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:26:57.226880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.130:38049 (local address 127.16.43.190:36415)
0620 09:26:57.226976 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:57.226978 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:57.227011 (+ 33us) server_negotiation.cc:407] Connection header received
0620 09:26:57.227095 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:57.227096 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:57.227108 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:57.227216 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:57.227572 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:57.228077 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:57.228800 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:57.228999 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:57.229087 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:57.232158 (+ 3071us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.43.130:38049: BlockingRecv error: recv got EOF from 127.16.43.130:38049 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 09:26:57.263458 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 25301
I20260620 09:26:57.267912 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 25452
I20260620 09:26:57.272290 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 25585
I20260620 09:26:57.276538 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 25230
2026-06-20T09:26:57Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17474 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-20T09:26:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:26:57Z Disabled control of system clock
I20260620 09:26:57.305043 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781947562949448-16558-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781947562949448-16558-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781947562949448-16558-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.43.190:43441
--webserver_interface=127.16.43.190
--webserver_port=0
--builtin_ntp_servers=127.16.43.148:39815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.43.190:43441
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 09:26:57.390213 25782 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:26:57.390424 25782 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:26:57.390443 25782 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:26:57.391757 25782 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:26:57.391799 25782 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:26:57.391812 25782 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:26:57.391825 25782 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:26:57.391837 25782 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:26:57.393124 25782 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:39815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781947562949448-16558-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781947562949448-16558-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.43.190:43441
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.43.190:43441
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.43.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25782
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781947562949448-16558-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:26:57.393326 25782 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:26:57.393570 25782 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:26:57.399389 25788 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:26:57.399549 25787 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:26:57.399472 25782 server_base.cc:1034] running on GCE node
W20260620 09:26:57.399506 25790 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:26:57.399860 25782 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:26:57.400194 25782 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:26:57.401396 25782 hybrid_clock.cc:648] HybridClock initialized: now 1781947617401374 us; error 58 us; skew 500 ppm
I20260620 09:26:57.402696 25782 webserver.cc:460] Webserver started at http://127.16.43.190:41201/ using document root <none> and password file <none>
I20260620 09:26:57.402951 25782 fs_manager.cc:359] Metadata directory not provided
I20260620 09:26:57.403103 25782 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:26:57.403259 25782 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:26:57.404453 25782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781947562949448-16558-0/minicluster-data/master-0/data/instance:
uuid: "ea9002f68cb44c619a8666cba69c0476"
format_stamp: "Formatted at 2026-06-20 09:26:57 on dist-test-slave-706q"
server_key: "d7f5b227afb18d8619b9b5c8cc8591a7"
server_key_iv: "6524bc9e2f3add8b761e4b8c74cd0080"
server_key_version: "encryptionkey@0"
I20260620 09:26:57.404845 25782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781947562949448-16558-0/minicluster-data/master-0/wal/instance:
uuid: "ea9002f68cb44c619a8666cba69c0476"
format_stamp: "Formatted at 2026-06-20 09:26:57 on dist-test-slave-706q"
server_key: "d7f5b227afb18d8619b9b5c8cc8591a7"
server_key_iv: "6524bc9e2f3add8b761e4b8c74cd0080"
server_key_version: "encryptionkey@0"
I20260620 09:26:57.406284 25782 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 09:26:57.407024 25796 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:57.407194 25782 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 09:26:57.407307 25782 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781947562949448-16558-0/minicluster-data/master-0/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781947562949448-16558-0/minicluster-data/master-0/wal
uuid: "ea9002f68cb44c619a8666cba69c0476"
format_stamp: "Formatted at 2026-06-20 09:26:57 on dist-test-slave-706q"
server_key: "d7f5b227afb18d8619b9b5c8cc8591a7"
server_key_iv: "6524bc9e2f3add8b761e4b8c74cd0080"
server_key_version: "encryptionkey@0"
I20260620 09:26:57.407387 25782 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781947562949448-16558-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781947562949448-16558-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781947562949448-16558-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:26:57.417057 25782 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:26:57.417425 25782 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:26:57.417591 25782 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:26:57.421913 25782 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.190:43441
I20260620 09:26:57.421948 25848 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.190:43441 every 8 connection(s)
I20260620 09:26:57.422314 25782 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
I20260620 09:26:57.423013 25849 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:26:57.425841 25849 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea9002f68cb44c619a8666cba69c0476: Bootstrap starting.
I20260620 09:26:57.426487 25849 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ea9002f68cb44c619a8666cba69c0476: Neither blocks nor log segments found. Creating new log.
I20260620 09:26:57.426813 25849 log.cc:824] T 00000000000000000000000000000000 P ea9002f68cb44c619a8666cba69c0476: Log is configured to *not* fsync() on all Append() calls
I20260620 09:26:57.427693 25849 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea9002f68cb44c619a8666cba69c0476: No bootstrap required, opened a new log
I20260620 09:26:57.429325 25849 raft_consensus.cc:348] T 00000000000000000000000000000000 P ea9002f68cb44c619a8666cba69c0476 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea9002f68cb44c619a8666cba69c0476" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 43441 } }
I20260620 09:26:57.429476 25849 raft_consensus.cc:374] T 00000000000000000000000000000000 P ea9002f68cb44c619a8666cba69c0476 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:26:57.429500 25849 raft_consensus.cc:729] T 00000000000000000000000000000000 P ea9002f68cb44c619a8666cba69c0476 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea9002f68cb44c619a8666cba69c0476, State: Initialized, Role: FOLLOWER
I20260620 09:26:57.429710 25849 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea9002f68cb44c619a8666cba69c0476 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea9002f68cb44c619a8666cba69c0476" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 43441 } }
I20260620 09:26:57.429800 25849 raft_consensus.cc:388] T 00000000000000000000000000000000 P ea9002f68cb44c619a8666cba69c0476 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:26:57.429859 25849 raft_consensus.cc:482] T 00000000000000000000000000000000 P ea9002f68cb44c619a8666cba69c0476 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:26:57.429837 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 25782
I20260620 09:26:57.429911 25849 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ea9002f68cb44c619a8666cba69c0476 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:26:57.429926 16558 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781947562949448-16558-0/minicluster-data/master-0/wal/instance
I20260620 09:26:57.430109 16558 external_mini_cluster.cc:1428] Setting key fddf980d859ba7ac33939fe2e6afbb8d
I20260620 09:26:57.430814 25849 raft_consensus.cc:504] T 00000000000000000000000000000000 P ea9002f68cb44c619a8666cba69c0476 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea9002f68cb44c619a8666cba69c0476" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 43441 } }
I20260620 09:26:57.430945 25849 leader_election.cc:302] T 00000000000000000000000000000000 P ea9002f68cb44c619a8666cba69c0476 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ea9002f68cb44c619a8666cba69c0476; no voters:
I20260620 09:26:57.431116 25849 leader_election.cc:288] T 00000000000000000000000000000000 P ea9002f68cb44c619a8666cba69c0476 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:26:57.431181 25854 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ea9002f68cb44c619a8666cba69c0476 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:26:57.431314 25854 raft_consensus.cc:686] T 00000000000000000000000000000000 P ea9002f68cb44c619a8666cba69c0476 [term 1 LEADER]: Becoming Leader. State: Replica: ea9002f68cb44c619a8666cba69c0476, State: Running, Role: LEADER
I20260620 09:26:57.431407 25849 sys_catalog.cc:564] T 00000000000000000000000000000000 P ea9002f68cb44c619a8666cba69c0476 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:26:57.431450 25854 consensus_queue.cc:237] T 00000000000000000000000000000000 P ea9002f68cb44c619a8666cba69c0476 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea9002f68cb44c619a8666cba69c0476" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 43441 } }
I20260620 09:26:57.431879 25854 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea9002f68cb44c619a8666cba69c0476 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ea9002f68cb44c619a8666cba69c0476. Latest consensus state: current_term: 1 leader_uuid: "ea9002f68cb44c619a8666cba69c0476" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea9002f68cb44c619a8666cba69c0476" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 43441 } } }
I20260620 09:26:57.431990 25854 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea9002f68cb44c619a8666cba69c0476 [sys.catalog]: This master's current role is: LEADER
I20260620 09:26:57.431887 25855 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea9002f68cb44c619a8666cba69c0476 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ea9002f68cb44c619a8666cba69c0476" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea9002f68cb44c619a8666cba69c0476" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 43441 } } }
I20260620 09:26:57.432195 25855 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea9002f68cb44c619a8666cba69c0476 [sys.catalog]: This master's current role is: LEADER
W20260620 09:26:57.433265 25869 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ea9002f68cb44c619a8666cba69c0476: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 09:26:57.433380 25869 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 09:26:57.433730 25862 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:26:57.433929 25862 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:26:57.435459 25862 catalog_manager.cc:1347] Generated new cluster ID: c96998db9d2242f0b82370a5a42383b3
I20260620 09:26:57.435519 25862 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:26:57.436790 25853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:57.430757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37710 (local address 127.16.43.190:43441)
0620 09:26:57.431011 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:57.431018 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:57.431072 (+ 54us) server_negotiation.cc:407] Connection header received
0620 09:26:57.431219 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:57.431227 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:57.431311 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:57.431481 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:57.431725 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:57.433360 (+ 1635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:57.434487 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:57.435080 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:57.435205 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:57.436082 (+ 877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:57.436086 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:57.436086 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:26:57.436263 (+ 177us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:57.436297 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:26:57.436309 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:57.436385 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:57.436486 (+ 101us) server_negotiation.cc:299] Negotiation successful
0620 09:26:57.436572 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":114,"threads_started":1}
I20260620 09:26:57.451112 25862 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:26:57.451897 25862 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:26:57.457134 25862 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ea9002f68cb44c619a8666cba69c0476: Generated new TSK 0
I20260620 09:26:57.457350 25862 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 09:26:57.460850 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781947562949448-16558-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781947562949448-16558-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781947562949448-16558-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781947562949448-16558-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.43.129:0
--local_ip_for_outbound_sockets=127.16.43.129
--webserver_interface=127.16.43.129
--webserver_port=0
--tserver_master_addrs=127.16.43.190:43441
--builtin_ntp_servers=127.16.43.148:39815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260620 09:26:57.564832 25873 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:26:57.565115 25873 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:26:57.565158 25873 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:26:57.566463 25873 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:26:57.566541 25873 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:26:57.566586 25873 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.43.129
I20260620 09:26:57.568286 25873 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:39815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781947562949448-16558-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781947562949448-16558-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.43.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781947562949448-16558-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.43.129
--webserver_port=0
--tserver_master_addrs=127.16.43.190:43441
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25873
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.43.129
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781947562949448-16558-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:26:57.568576 25873 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:26:57.568898 25873 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:26:57.574887 25879 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:26:57.574940 25881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:26:57.574898 25878 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:26:57.575327 25873 server_base.cc:1034] running on GCE node
I20260620 09:26:57.575595 25873 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:26:57.575928 25873 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:26:57.577101 25873 hybrid_clock.cc:648] HybridClock initialized: now 1781947617577088 us; error 31 us; skew 500 ppm
I20260620 09:26:57.578454 25873 webserver.cc:460] Webserver started at http://127.16.43.129:33303/ using document root <none> and password file <none>
I20260620 09:26:57.578759 25873 fs_manager.cc:359] Metadata directory not provided
I20260620 09:26:57.578832 25873 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:26:57.578985 25873 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:26:57.580178 25873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781947562949448-16558-0/minicluster-data/ts-0/data/instance:
uuid: "ef5ffbc7024144f29d753c1ef92366da"
format_stamp: "Formatted at 2026-06-20 09:26:57 on dist-test-slave-706q"
server_key: "951f2809b51484c01ed96d53f096147b"
server_key_iv: "4ac72fbd45a8398f565910e9721d4c18"
server_key_version: "encryptionkey@0"
I20260620 09:26:57.580564 25873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781947562949448-16558-0/minicluster-data/ts-0/wal/instance:
uuid: "ef5ffbc7024144f29d753c1ef92366da"
format_stamp: "Formatted at 2026-06-20 09:26:57 on dist-test-slave-706q"
server_key: "951f2809b51484c01ed96d53f096147b"
server_key_iv: "4ac72fbd45a8398f565910e9721d4c18"
server_key_version: "encryptionkey@0"
I20260620 09:26:57.582020 25873 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 09:26:57.582840 25887 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:57.583190 25873 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:26:57.583299 25873 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781947562949448-16558-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781947562949448-16558-0/minicluster-data/ts-0/wal
uuid: "ef5ffbc7024144f29d753c1ef92366da"
format_stamp: "Formatted at 2026-06-20 09:26:57 on dist-test-slave-706q"
server_key: "951f2809b51484c01ed96d53f096147b"
server_key_iv: "4ac72fbd45a8398f565910e9721d4c18"
server_key_version: "encryptionkey@0"
I20260620 09:26:57.583387 25873 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781947562949448-16558-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781947562949448-16558-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781947562949448-16558-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:26:57.596134 25873 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:26:57.596474 25873 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:26:57.596633 25873 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:26:57.596865 25873 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:26:57.597256 25873 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:26:57.597314 25873 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:57.597350 25873 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:26:57.597379 25873 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:57.603220 25873 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.129:42215
I20260620 09:26:57.603317 26000 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.129:42215 every 8 connection(s)
I20260620 09:26:57.603720 25873 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781947562949448-16558-0/minicluster-data/ts-0/data/info.pb
I20260620 09:26:57.607074 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 25873
I20260620 09:26:57.607156 16558 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781947562949448-16558-0/minicluster-data/ts-0/wal/instance
I20260620 09:26:57.607378 16558 external_mini_cluster.cc:1428] Setting key bf3502239f3eaeea34f34779dabc3e51
I20260620 09:26:57.608327 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781947562949448-16558-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781947562949448-16558-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781947562949448-16558-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781947562949448-16558-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.43.130:0
--local_ip_for_outbound_sockets=127.16.43.130
--webserver_interface=127.16.43.130
--webserver_port=0
--tserver_master_addrs=127.16.43.190:43441
--builtin_ntp_servers=127.16.43.148:39815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 09:26:57.609261 25853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:57.604512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.129:40625 (local address 127.16.43.190:43441)
0620 09:26:57.604597 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:57.604600 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:57.604801 (+ 201us) server_negotiation.cc:407] Connection header received
0620 09:26:57.605278 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:57.605279 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:57.605295 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:57.605347 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:57.606028 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:57.606520 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:57.607741 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:57.607941 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:57.608029 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:57.608963 (+ 934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:57.608964 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:57.608964 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:26:57.608972 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:57.608991 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:26:57.608995 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:57.609058 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:57.609144 (+ 86us) server_negotiation.cc:299] Negotiation successful
0620 09:26:57.609183 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 09:26:57.609480 26003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:57.604475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:43441 (local address 127.16.43.129:40625)
0620 09:26:57.604666 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:57.604708 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:57.605192 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:57.605405 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:57.605408 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:57.605417 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:57.605944 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:57.605948 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:57.606603 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:57.606603 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:57.607628 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:57.607631 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:57.608040 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:57.608041 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:57.608190 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:57.608813 (+ 623us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 09:26:57.608813 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:26:57.608844 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 09:26:57.608867 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:26:57.609057 (+ 190us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:26:57.609064 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:26:57.609068 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 09:26:57.609134 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 09:26:57.609239 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":99,"threads_started":1}
I20260620 09:26:57.610208 26001 heartbeater.cc:347] Connected to a master server at 127.16.43.190:43441
I20260620 09:26:57.610302 26001 heartbeater.cc:464] Registering TS with master...
I20260620 09:26:57.610513 26001 heartbeater.cc:511] Master 127.16.43.190:43441 requested a full tablet report, sending...
I20260620 09:26:57.611120 25813 ts_manager.cc:194] Registered new tserver with Master: ef5ffbc7024144f29d753c1ef92366da (127.16.43.129:42215)
I20260620 09:26:57.611836 25813 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.43.129:40625
W20260620 09:26:57.691879 26004 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:26:57.692148 26004 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:26:57.692188 26004 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:26:57.693414 26004 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:26:57.693487 26004 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:26:57.693531 26004 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.43.130
I20260620 09:26:57.694917 26004 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:39815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781947562949448-16558-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781947562949448-16558-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.43.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781947562949448-16558-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.43.130
--webserver_port=0
--tserver_master_addrs=127.16.43.190:43441
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26004
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.43.130
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781947562949448-16558-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:26:57.695204 26004 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:26:57.695560 26004 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:26:57.700836 26009 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:26:57.700937 26012 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:26:57.700979 26004 server_base.cc:1034] running on GCE node
W20260620 09:26:57.701050 26010 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:26:57.701386 26004 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:26:57.701650 26004 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:26:57.702834 26004 hybrid_clock.cc:648] HybridClock initialized: now 1781947617702833 us; error 37 us; skew 500 ppm
I20260620 09:26:57.704277 26004 webserver.cc:460] Webserver started at http://127.16.43.130:40591/ using document root <none> and password file <none>
I20260620 09:26:57.704540 26004 fs_manager.cc:359] Metadata directory not provided
I20260620 09:26:57.704612 26004 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:26:57.704775 26004 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:26:57.705952 26004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781947562949448-16558-0/minicluster-data/ts-1/data/instance:
uuid: "f9c218625c9f4674bc2c36163780b58d"
format_stamp: "Formatted at 2026-06-20 09:26:57 on dist-test-slave-706q"
server_key: "5c036ada83976cee5c3fc2138f6dcc67"
server_key_iv: "4cf27a49d39ab063b1fb6aa6d2d43638"
server_key_version: "encryptionkey@0"
I20260620 09:26:57.706346 26004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781947562949448-16558-0/minicluster-data/ts-1/wal/instance:
uuid: "f9c218625c9f4674bc2c36163780b58d"
format_stamp: "Formatted at 2026-06-20 09:26:57 on dist-test-slave-706q"
server_key: "5c036ada83976cee5c3fc2138f6dcc67"
server_key_iv: "4cf27a49d39ab063b1fb6aa6d2d43638"
server_key_version: "encryptionkey@0"
I20260620 09:26:57.707860 26004 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 09:26:57.708681 26018 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:57.708866 26004 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 09:26:57.708981 26004 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781947562949448-16558-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781947562949448-16558-0/minicluster-data/ts-1/wal
uuid: "f9c218625c9f4674bc2c36163780b58d"
format_stamp: "Formatted at 2026-06-20 09:26:57 on dist-test-slave-706q"
server_key: "5c036ada83976cee5c3fc2138f6dcc67"
server_key_iv: "4cf27a49d39ab063b1fb6aa6d2d43638"
server_key_version: "encryptionkey@0"
I20260620 09:26:57.709061 26004 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781947562949448-16558-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781947562949448-16558-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781947562949448-16558-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:26:57.719221 26004 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:26:57.719586 26004 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:26:57.719755 26004 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:26:57.720000 26004 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:26:57.720336 26004 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:26:57.720392 26004 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:57.720436 26004 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:26:57.720461 26004 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:57.726449 26004 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.130:38431
I20260620 09:26:57.726526 26131 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.130:38431 every 8 connection(s)
I20260620 09:26:57.726892 26004 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781947562949448-16558-0/minicluster-data/ts-1/data/info.pb
I20260620 09:26:57.732753 25853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:57.727801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.130:42017 (local address 127.16.43.190:43441)
0620 09:26:57.727847 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:57.727849 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:57.728115 (+ 266us) server_negotiation.cc:407] Connection header received
0620 09:26:57.728664 (+ 549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:57.728665 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:57.728676 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:57.728736 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:57.729437 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:57.729989 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:57.731000 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:57.731261 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:57.731324 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:57.732343 (+ 1019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:57.732344 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:57.732344 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:26:57.732350 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:57.732365 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:26:57.732369 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:57.732419 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:57.732654 (+ 235us) server_negotiation.cc:299] Negotiation successful
0620 09:26:57.732683 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 09:26:57.732941 26134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:57.727800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:43441 (local address 127.16.43.130:42017)
0620 09:26:57.727988 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:57.728023 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:57.728544 (+ 521us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:57.728792 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:57.728795 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:57.728803 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:57.729319 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:57.729323 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:57.730097 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:57.730099 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:57.730912 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:57.730914 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:57.731345 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:57.731346 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:57.731516 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:57.732148 (+ 632us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 09:26:57.732148 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:26:57.732194 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 09:26:57.732240 (+ 46us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:26:57.732526 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:26:57.732534 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:26:57.732537 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 09:26:57.732620 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 09:26:57.732709 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":94,"threads_started":1}
I20260620 09:26:57.733613 26132 heartbeater.cc:347] Connected to a master server at 127.16.43.190:43441
I20260620 09:26:57.733742 26132 heartbeater.cc:464] Registering TS with master...
I20260620 09:26:57.733989 26132 heartbeater.cc:511] Master 127.16.43.190:43441 requested a full tablet report, sending...
I20260620 09:26:57.734030 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 26004
I20260620 09:26:57.734144 16558 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781947562949448-16558-0/minicluster-data/ts-1/wal/instance
I20260620 09:26:57.734407 16558 external_mini_cluster.cc:1428] Setting key 762940f0a9bd46c47615e839a547e64d
I20260620 09:26:57.734495 25813 ts_manager.cc:194] Registered new tserver with Master: f9c218625c9f4674bc2c36163780b58d (127.16.43.130:38431)
I20260620 09:26:57.735107 25813 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.43.130:42017
I20260620 09:26:57.735478 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781947562949448-16558-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781947562949448-16558-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781947562949448-16558-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781947562949448-16558-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.43.131:0
--local_ip_for_outbound_sockets=127.16.43.131
--webserver_interface=127.16.43.131
--webserver_port=0
--tserver_master_addrs=127.16.43.190:43441
--builtin_ntp_servers=127.16.43.148:39815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260620 09:26:57.822546 26135 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:26:57.822777 26135 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:26:57.822800 26135 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:26:57.824040 26135 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:26:57.824088 26135 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:26:57.824119 26135 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.43.131
I20260620 09:26:57.825477 26135 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:39815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781947562949448-16558-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781947562949448-16558-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.43.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781947562949448-16558-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.43.131
--webserver_port=0
--tserver_master_addrs=127.16.43.190:43441
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26135
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.43.131
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781947562949448-16558-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:26:57.825770 26135 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:26:57.826089 26135 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:26:57.831498 26141 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:26:57.831482 26143 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:26:57.831689 26135 server_base.cc:1034] running on GCE node
W20260620 09:26:57.831482 26140 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:26:57.832104 26135 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:26:57.832379 26135 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:26:57.833549 26135 hybrid_clock.cc:648] HybridClock initialized: now 1781947617833514 us; error 53 us; skew 500 ppm
I20260620 09:26:57.834956 26135 webserver.cc:460] Webserver started at http://127.16.43.131:42051/ using document root <none> and password file <none>
I20260620 09:26:57.835160 26135 fs_manager.cc:359] Metadata directory not provided
I20260620 09:26:57.835203 26135 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:26:57.835294 26135 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:26:57.836499 26135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781947562949448-16558-0/minicluster-data/ts-2/data/instance:
uuid: "65a86429c0ab4c72b2b8e80696fc109c"
format_stamp: "Formatted at 2026-06-20 09:26:57 on dist-test-slave-706q"
server_key: "c8cf9f4736d10043dd5fe706d5cd64db"
server_key_iv: "2bf31a78be6f518a2f0152d6d9cd5778"
server_key_version: "encryptionkey@0"
I20260620 09:26:57.836813 26135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781947562949448-16558-0/minicluster-data/ts-2/wal/instance:
uuid: "65a86429c0ab4c72b2b8e80696fc109c"
format_stamp: "Formatted at 2026-06-20 09:26:57 on dist-test-slave-706q"
server_key: "c8cf9f4736d10043dd5fe706d5cd64db"
server_key_iv: "2bf31a78be6f518a2f0152d6d9cd5778"
server_key_version: "encryptionkey@0"
I20260620 09:26:57.838124 26135 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 09:26:57.838891 26149 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:57.839118 26135 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 09:26:57.839282 26135 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781947562949448-16558-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781947562949448-16558-0/minicluster-data/ts-2/wal
uuid: "65a86429c0ab4c72b2b8e80696fc109c"
format_stamp: "Formatted at 2026-06-20 09:26:57 on dist-test-slave-706q"
server_key: "c8cf9f4736d10043dd5fe706d5cd64db"
server_key_iv: "2bf31a78be6f518a2f0152d6d9cd5778"
server_key_version: "encryptionkey@0"
I20260620 09:26:57.839385 26135 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781947562949448-16558-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781947562949448-16558-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781947562949448-16558-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:26:57.851274 26135 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:26:57.851681 26135 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:26:57.851845 26135 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:26:57.852110 26135 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:26:57.852473 26135 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:26:57.852530 26135 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:57.852574 26135 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:26:57.852603 26135 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:57.858636 26135 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.131:34803
I20260620 09:26:57.858686 26262 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.131:34803 every 8 connection(s)
I20260620 09:26:57.859032 26135 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781947562949448-16558-0/minicluster-data/ts-2/data/info.pb
I20260620 09:26:57.860520 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 26135
I20260620 09:26:57.860633 16558 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781947562949448-16558-0/minicluster-data/ts-2/wal/instance
I20260620 09:26:57.860862 16558 external_mini_cluster.cc:1428] Setting key e2e5b56d1cfb2a69f775cd2cffe74ef1
I20260620 09:26:57.864696 25853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:57.859851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.131:33773 (local address 127.16.43.190:43441)
0620 09:26:57.859937 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:57.859940 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:57.860171 (+ 231us) server_negotiation.cc:407] Connection header received
0620 09:26:57.860770 (+ 599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:57.860772 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:57.860785 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:57.860838 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:57.861948 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:57.862461 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:57.863428 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:57.863683 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:57.863737 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:57.864468 (+ 731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:57.864470 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:57.864470 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:26:57.864474 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:57.864489 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:26:57.864492 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:57.864538 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:57.864606 (+ 68us) server_negotiation.cc:299] Negotiation successful
0620 09:26:57.864631 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 09:26:57.864933 26265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:57.859808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:43441 (local address 127.16.43.131:33773)
0620 09:26:57.860028 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:57.860081 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:57.860571 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:57.860956 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:57.860962 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:57.861043 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:57.861741 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:57.861749 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:57.862547 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:57.862548 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:57.863349 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:57.863351 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:57.863745 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:57.863746 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:26:57.863862 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:57.864340 (+ 478us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 09:26:57.864341 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:26:57.864369 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 09:26:57.864391 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:26:57.864541 (+ 150us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:26:57.864544 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:26:57.864546 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 09:26:57.864594 (+ 48us) client_negotiation.cc:240] Negotiation successful
0620 09:26:57.864671 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":101,"threads_started":1}
I20260620 09:26:57.865430 26263 heartbeater.cc:347] Connected to a master server at 127.16.43.190:43441
I20260620 09:26:57.865540 26263 heartbeater.cc:464] Registering TS with master...
I20260620 09:26:57.865748 26263 heartbeater.cc:511] Master 127.16.43.190:43441 requested a full tablet report, sending...
I20260620 09:26:57.866144 25813 ts_manager.cc:194] Registered new tserver with Master: 65a86429c0ab4c72b2b8e80696fc109c (127.16.43.131:34803)
I20260620 09:26:57.866508 25813 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.43.131:33773
I20260620 09:26:57.873041 16558 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 09:26:57.874987 25853 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:26:57.874464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37724 (local address 127.16.43.190:43441)
0620 09:26:57.874554 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:57.874561 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:57.874659 (+ 98us) server_negotiation.cc:407] Connection header received
0620 09:26:57.874687 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:57.874687 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:57.874699 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:57.874752 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:57.874903 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37724: BlockingRecv error: recv got EOF from 127.0.0.1:37724 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 09:26:57.875142 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 25873
I20260620 09:26:57.879907 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 26004
I20260620 09:26:57.884682 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 26135
I20260620 09:26:57.889273 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 25782
2026-06-20T09:26:57Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (612 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-20T09:26:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:26:57Z Disabled control of system clock
I20260620 09:26:57.916937 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781947562949448-16558-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781947562949448-16558-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781947562949448-16558-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.43.190:46261
--webserver_interface=127.16.43.190
--webserver_port=0
--builtin_ntp_servers=127.16.43.148:34653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.43.190:46261
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 09:26:57.998536 26274 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:26:57.998745 26274 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:26:57.998766 26274 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:26:57.999861 26274 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:26:57.999894 26274 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:26:57.999907 26274 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:26:57.999919 26274 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:26:57.999931 26274 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:26:58.001185 26274 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:34653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781947562949448-16558-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781947562949448-16558-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.43.190:46261
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.43.190:46261
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.43.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26274
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781947562949448-16558-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:26:58.001379 26274 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:26:58.001621 26274 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:26:58.006767 26282 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:26:58.006767 26279 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:26:58.007061 26274 server_base.cc:1034] running on GCE node
W20260620 09:26:58.006803 26280 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:26:58.007371 26274 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:26:58.007700 26274 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:26:58.008888 26274 hybrid_clock.cc:648] HybridClock initialized: now 1781947618008829 us; error 46 us; skew 500 ppm
I20260620 09:26:58.010147 26274 webserver.cc:460] Webserver started at http://127.16.43.190:46485/ using document root <none> and password file <none>
I20260620 09:26:58.010370 26274 fs_manager.cc:359] Metadata directory not provided
I20260620 09:26:58.010411 26274 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:26:58.010501 26274 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:26:58.011627 26274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781947562949448-16558-0/minicluster-data/master-0/data/instance:
uuid: "26afc05504d44e0a83128701f776fd6b"
format_stamp: "Formatted at 2026-06-20 09:26:58 on dist-test-slave-706q"
server_key: "a9a15d897708973349a575a104aae5bb"
server_key_iv: "05a2710508f55af834ed231967fa6f5e"
server_key_version: "encryptionkey@0"
I20260620 09:26:58.011979 26274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781947562949448-16558-0/minicluster-data/master-0/wal/instance:
uuid: "26afc05504d44e0a83128701f776fd6b"
format_stamp: "Formatted at 2026-06-20 09:26:58 on dist-test-slave-706q"
server_key: "a9a15d897708973349a575a104aae5bb"
server_key_iv: "05a2710508f55af834ed231967fa6f5e"
server_key_version: "encryptionkey@0"
I20260620 09:26:58.013294 26274 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 09:26:58.014014 26288 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:58.014230 26274 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 09:26:58.014343 26274 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781947562949448-16558-0/minicluster-data/master-0/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781947562949448-16558-0/minicluster-data/master-0/wal
uuid: "26afc05504d44e0a83128701f776fd6b"
format_stamp: "Formatted at 2026-06-20 09:26:58 on dist-test-slave-706q"
server_key: "a9a15d897708973349a575a104aae5bb"
server_key_iv: "05a2710508f55af834ed231967fa6f5e"
server_key_version: "encryptionkey@0"
I20260620 09:26:58.014425 26274 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781947562949448-16558-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781947562949448-16558-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781947562949448-16558-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:26:58.017990 26274 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:26:58.018298 26274 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:26:58.018458 26274 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:26:58.022608 26274 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.190:46261
I20260620 09:26:58.022639 26340 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.190:46261 every 8 connection(s)
I20260620 09:26:58.022995 26274 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
I20260620 09:26:58.023713 26341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:26:58.026571 26341 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26afc05504d44e0a83128701f776fd6b: Bootstrap starting.
I20260620 09:26:58.027196 26341 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 26afc05504d44e0a83128701f776fd6b: Neither blocks nor log segments found. Creating new log.
I20260620 09:26:58.027500 26341 log.cc:824] T 00000000000000000000000000000000 P 26afc05504d44e0a83128701f776fd6b: Log is configured to *not* fsync() on all Append() calls
I20260620 09:26:58.028302 26341 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26afc05504d44e0a83128701f776fd6b: No bootstrap required, opened a new log
I20260620 09:26:58.029886 26341 raft_consensus.cc:348] T 00000000000000000000000000000000 P 26afc05504d44e0a83128701f776fd6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26afc05504d44e0a83128701f776fd6b" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 46261 } }
I20260620 09:26:58.030033 26341 raft_consensus.cc:374] T 00000000000000000000000000000000 P 26afc05504d44e0a83128701f776fd6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:26:58.030055 26341 raft_consensus.cc:729] T 00000000000000000000000000000000 P 26afc05504d44e0a83128701f776fd6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26afc05504d44e0a83128701f776fd6b, State: Initialized, Role: FOLLOWER
I20260620 09:26:58.030251 26341 consensus_queue.cc:260] T 00000000000000000000000000000000 P 26afc05504d44e0a83128701f776fd6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26afc05504d44e0a83128701f776fd6b" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 46261 } }
I20260620 09:26:58.030314 26341 raft_consensus.cc:388] T 00000000000000000000000000000000 P 26afc05504d44e0a83128701f776fd6b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:26:58.030360 26341 raft_consensus.cc:482] T 00000000000000000000000000000000 P 26afc05504d44e0a83128701f776fd6b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:26:58.030409 26341 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 26afc05504d44e0a83128701f776fd6b [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:26:58.030967 26341 raft_consensus.cc:504] T 00000000000000000000000000000000 P 26afc05504d44e0a83128701f776fd6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26afc05504d44e0a83128701f776fd6b" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 46261 } }
I20260620 09:26:58.031059 26341 leader_election.cc:302] T 00000000000000000000000000000000 P 26afc05504d44e0a83128701f776fd6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 26afc05504d44e0a83128701f776fd6b; no voters:
I20260620 09:26:58.031153 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 26274
I20260620 09:26:58.031262 16558 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781947562949448-16558-0/minicluster-data/master-0/wal/instance
I20260620 09:26:58.031265 26341 leader_election.cc:288] T 00000000000000000000000000000000 P 26afc05504d44e0a83128701f776fd6b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:26:58.031306 26344 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 26afc05504d44e0a83128701f776fd6b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:26:58.031468 16558 external_mini_cluster.cc:1428] Setting key 838b77a35d22bd19638f5f8b2e80cf91
I20260620 09:26:58.031467 26344 raft_consensus.cc:686] T 00000000000000000000000000000000 P 26afc05504d44e0a83128701f776fd6b [term 1 LEADER]: Becoming Leader. State: Replica: 26afc05504d44e0a83128701f776fd6b, State: Running, Role: LEADER
I20260620 09:26:58.031579 26341 sys_catalog.cc:564] T 00000000000000000000000000000000 P 26afc05504d44e0a83128701f776fd6b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:26:58.031581 26344 consensus_queue.cc:237] T 00000000000000000000000000000000 P 26afc05504d44e0a83128701f776fd6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26afc05504d44e0a83128701f776fd6b" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 46261 } }
I20260620 09:26:58.031972 26344 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26afc05504d44e0a83128701f776fd6b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 26afc05504d44e0a83128701f776fd6b. Latest consensus state: current_term: 1 leader_uuid: "26afc05504d44e0a83128701f776fd6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26afc05504d44e0a83128701f776fd6b" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 46261 } } }
I20260620 09:26:58.032068 26344 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26afc05504d44e0a83128701f776fd6b [sys.catalog]: This master's current role is: LEADER
I20260620 09:26:58.031982 26345 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26afc05504d44e0a83128701f776fd6b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "26afc05504d44e0a83128701f776fd6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26afc05504d44e0a83128701f776fd6b" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 46261 } } }
I20260620 09:26:58.032264 26345 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26afc05504d44e0a83128701f776fd6b [sys.catalog]: This master's current role is: LEADER
W20260620 09:26:58.033911 26362 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 26afc05504d44e0a83128701f776fd6b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 09:26:58.033970 26362 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 09:26:58.034264 26355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:58.032375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44292 (local address 127.16.43.190:46261)
0620 09:26:58.032577 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:58.032588 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:58.032619 (+ 31us) server_negotiation.cc:407] Connection header received
0620 09:26:58.032792 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:58.032800 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:58.032853 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:58.033000 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:58.033637 (+ 637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:58.033642 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:58.033642 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:26:58.033651 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:58.033714 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:26:58.033732 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:58.033801 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:58.033930 (+ 129us) server_negotiation.cc:299] Negotiation successful
0620 09:26:58.034087 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":79,"threads_started":1}
I20260620 09:26:58.034370 26352 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:26:58.034547 26352 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:26:58.036136 26352 catalog_manager.cc:1347] Generated new cluster ID: 60b1200afabb4cc6a3f16f35832d211e
I20260620 09:26:58.036224 26352 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:26:58.044795 26352 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:26:58.044981 26352 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:26:58.050338 26352 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 26afc05504d44e0a83128701f776fd6b: Generated new TSK 0
I20260620 09:26:58.050555 26352 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 09:26:58.060983 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781947562949448-16558-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781947562949448-16558-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781947562949448-16558-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781947562949448-16558-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.43.129:0
--local_ip_for_outbound_sockets=127.16.43.129
--webserver_interface=127.16.43.129
--webserver_port=0
--tserver_master_addrs=127.16.43.190:46261
--builtin_ntp_servers=127.16.43.148:34653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 09:26:58.154155 26365 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:26:58.154418 26365 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:26:58.154459 26365 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:26:58.155759 26365 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:26:58.155821 26365 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:26:58.155869 26365 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.43.129
I20260620 09:26:58.157366 26365 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:34653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781947562949448-16558-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781947562949448-16558-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.43.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781947562949448-16558-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.43.129
--webserver_port=0
--tserver_master_addrs=127.16.43.190:46261
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26365
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.43.129
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781947562949448-16558-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:26:58.157636 26365 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:26:58.157948 26365 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:26:58.164003 26371 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:26:58.164022 26373 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:26:58.164283 26370 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:26:58.164402 26365 server_base.cc:1034] running on GCE node
I20260620 09:26:58.164619 26365 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:26:58.164904 26365 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:26:58.166092 26365 hybrid_clock.cc:648] HybridClock initialized: now 1781947618166078 us; error 30 us; skew 500 ppm
I20260620 09:26:58.167510 26365 webserver.cc:460] Webserver started at http://127.16.43.129:40821/ using document root <none> and password file <none>
I20260620 09:26:58.167824 26365 fs_manager.cc:359] Metadata directory not provided
I20260620 09:26:58.167901 26365 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:26:58.168056 26365 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:26:58.169193 26365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781947562949448-16558-0/minicluster-data/ts-0/data/instance:
uuid: "7d3869be54254ef6a9c43e37256c2339"
format_stamp: "Formatted at 2026-06-20 09:26:58 on dist-test-slave-706q"
server_key: "78cd9c082054a557567e0f27ffbae4d4"
server_key_iv: "4bc685321510507dc2516a44b9a63b5d"
server_key_version: "encryptionkey@0"
I20260620 09:26:58.169570 26365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781947562949448-16558-0/minicluster-data/ts-0/wal/instance:
uuid: "7d3869be54254ef6a9c43e37256c2339"
format_stamp: "Formatted at 2026-06-20 09:26:58 on dist-test-slave-706q"
server_key: "78cd9c082054a557567e0f27ffbae4d4"
server_key_iv: "4bc685321510507dc2516a44b9a63b5d"
server_key_version: "encryptionkey@0"
I20260620 09:26:58.171031 26365 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 09:26:58.171943 26379 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:58.172117 26365 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:26:58.172256 26365 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781947562949448-16558-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781947562949448-16558-0/minicluster-data/ts-0/wal
uuid: "7d3869be54254ef6a9c43e37256c2339"
format_stamp: "Formatted at 2026-06-20 09:26:58 on dist-test-slave-706q"
server_key: "78cd9c082054a557567e0f27ffbae4d4"
server_key_iv: "4bc685321510507dc2516a44b9a63b5d"
server_key_version: "encryptionkey@0"
I20260620 09:26:58.172339 26365 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781947562949448-16558-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781947562949448-16558-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781947562949448-16558-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:26:58.175724 26365 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:26:58.176020 26365 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:26:58.176179 26365 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:26:58.176422 26365 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:26:58.176823 26365 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:26:58.176867 26365 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:58.176913 26365 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:26:58.176937 26365 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:58.183655 26365 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.129:34643
I20260620 09:26:58.183727 26492 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.129:34643 every 8 connection(s)
I20260620 09:26:58.184077 26365 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781947562949448-16558-0/minicluster-data/ts-0/data/info.pb
I20260620 09:26:58.187134 26355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:58.184872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.129:46449 (local address 127.16.43.190:46261)
0620 09:26:58.184918 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:58.184921 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:58.185202 (+ 281us) server_negotiation.cc:407] Connection header received
0620 09:26:58.185628 (+ 426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:58.185629 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:58.185634 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:58.185700 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:58.186856 (+ 1156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:58.186858 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:58.186858 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:26:58.186862 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:58.186877 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:26:58.186881 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:58.186935 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:58.187066 (+ 131us) server_negotiation.cc:299] Negotiation successful
0620 09:26:58.187081 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 09:26:58.187355 26495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:58.184837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:46261 (local address 127.16.43.129:46449)
0620 09:26:58.185049 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:58.185091 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:58.185548 (+ 457us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:58.185774 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:58.185777 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:58.185786 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:58.186697 (+ 911us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 09:26:58.186697 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:26:58.186728 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 09:26:58.186758 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:26:58.186969 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:26:58.186975 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:26:58.186978 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 09:26:58.187060 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 09:26:58.187116 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":104,"threads_started":1}
I20260620 09:26:58.187923 26493 heartbeater.cc:347] Connected to a master server at 127.16.43.190:46261
I20260620 09:26:58.188019 26493 heartbeater.cc:464] Registering TS with master...
I20260620 09:26:58.188139 26493 heartbeater.cc:511] Master 127.16.43.190:46261 requested a full tablet report, sending...
I20260620 09:26:58.188422 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 26365
I20260620 09:26:58.188532 16558 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781947562949448-16558-0/minicluster-data/ts-0/wal/instance
I20260620 09:26:58.188740 16558 external_mini_cluster.cc:1428] Setting key 52e7b6220a7e8f7d7c54250dd590cefe
I20260620 09:26:58.188725 26305 ts_manager.cc:194] Registered new tserver with Master: 7d3869be54254ef6a9c43e37256c2339 (127.16.43.129:34643)
I20260620 09:26:58.189744 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781947562949448-16558-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781947562949448-16558-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781947562949448-16558-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781947562949448-16558-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.43.130:0
--local_ip_for_outbound_sockets=127.16.43.130
--webserver_interface=127.16.43.130
--webserver_port=0
--tserver_master_addrs=127.16.43.190:46261
--builtin_ntp_servers=127.16.43.148:34653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 09:26:58.272456 26496 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:26:58.272703 26496 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:26:58.272729 26496 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:26:58.274047 26496 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:26:58.274114 26496 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:26:58.274161 26496 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.43.130
I20260620 09:26:58.275732 26496 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:34653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781947562949448-16558-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781947562949448-16558-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.43.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781947562949448-16558-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.43.130
--webserver_port=0
--tserver_master_addrs=127.16.43.190:46261
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26496
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.43.130
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781947562949448-16558-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:26:58.276018 26496 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:26:58.276310 26496 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:26:58.281595 26502 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:26:58.281610 26504 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:26:58.281836 26496 server_base.cc:1034] running on GCE node
W20260620 09:26:58.281602 26501 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:26:58.282236 26496 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:26:58.282498 26496 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:26:58.283622 26496 hybrid_clock.cc:648] HybridClock initialized: now 1781947618283599 us; error 41 us; skew 500 ppm
I20260620 09:26:58.284978 26496 webserver.cc:460] Webserver started at http://127.16.43.130:35921/ using document root <none> and password file <none>
I20260620 09:26:58.285262 26496 fs_manager.cc:359] Metadata directory not provided
I20260620 09:26:58.285312 26496 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:26:58.285487 26496 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:26:58.286626 26496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781947562949448-16558-0/minicluster-data/ts-1/data/instance:
uuid: "e46edd3040844cc48cda08ed2c12d199"
format_stamp: "Formatted at 2026-06-20 09:26:58 on dist-test-slave-706q"
server_key: "fcd61554f4d79d0b4984c602d75ed77e"
server_key_iv: "fecd539249c8db3c8f6eddaa739436dd"
server_key_version: "encryptionkey@0"
I20260620 09:26:58.287020 26496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781947562949448-16558-0/minicluster-data/ts-1/wal/instance:
uuid: "e46edd3040844cc48cda08ed2c12d199"
format_stamp: "Formatted at 2026-06-20 09:26:58 on dist-test-slave-706q"
server_key: "fcd61554f4d79d0b4984c602d75ed77e"
server_key_iv: "fecd539249c8db3c8f6eddaa739436dd"
server_key_version: "encryptionkey@0"
I20260620 09:26:58.288513 26496 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 09:26:58.289397 26510 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:58.289633 26496 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:26:58.289736 26496 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781947562949448-16558-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781947562949448-16558-0/minicluster-data/ts-1/wal
uuid: "e46edd3040844cc48cda08ed2c12d199"
format_stamp: "Formatted at 2026-06-20 09:26:58 on dist-test-slave-706q"
server_key: "fcd61554f4d79d0b4984c602d75ed77e"
server_key_iv: "fecd539249c8db3c8f6eddaa739436dd"
server_key_version: "encryptionkey@0"
I20260620 09:26:58.289817 26496 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781947562949448-16558-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781947562949448-16558-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781947562949448-16558-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:26:58.293452 26496 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:26:58.293767 26496 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:26:58.293915 26496 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:26:58.294127 26496 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:26:58.294507 26496 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:26:58.294543 26496 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:58.294564 26496 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:26:58.294610 26496 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:58.300729 26496 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.130:32883
I20260620 09:26:58.300784 26623 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.130:32883 every 8 connection(s)
I20260620 09:26:58.301136 26496 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781947562949448-16558-0/minicluster-data/ts-1/data/info.pb
I20260620 09:26:58.304272 26355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:58.301895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.130:45983 (local address 127.16.43.190:46261)
0620 09:26:58.301958 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:58.301960 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:58.302213 (+ 253us) server_negotiation.cc:407] Connection header received
0620 09:26:58.302672 (+ 459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:58.302673 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:58.302679 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:58.302736 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:58.304066 (+ 1330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:58.304067 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:58.304067 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:26:58.304071 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:58.304085 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:26:58.304088 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:58.304136 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:58.304204 (+ 68us) server_negotiation.cc:299] Negotiation successful
0620 09:26:58.304217 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 09:26:58.304507 26626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:58.301882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:46261 (local address 127.16.43.130:45983)
0620 09:26:58.302071 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:58.302107 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:58.302583 (+ 476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:58.302798 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:58.302802 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:58.302810 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:58.303917 (+ 1107us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 09:26:58.303918 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:26:58.303950 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 09:26:58.303982 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:26:58.304135 (+ 153us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:26:58.304137 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:26:58.304139 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 09:26:58.304199 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 09:26:58.304252 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":99,"threads_started":1}
I20260620 09:26:58.304690 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 26496
I20260620 09:26:58.304791 16558 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781947562949448-16558-0/minicluster-data/ts-1/wal/instance
I20260620 09:26:58.304986 16558 external_mini_cluster.cc:1428] Setting key d6fc3f7edefdb72163aeec28fd74fd54
I20260620 09:26:58.305014 26624 heartbeater.cc:347] Connected to a master server at 127.16.43.190:46261
I20260620 09:26:58.305137 26624 heartbeater.cc:464] Registering TS with master...
I20260620 09:26:58.305342 26624 heartbeater.cc:511] Master 127.16.43.190:46261 requested a full tablet report, sending...
I20260620 09:26:58.305771 26305 ts_manager.cc:194] Registered new tserver with Master: e46edd3040844cc48cda08ed2c12d199 (127.16.43.130:32883)
I20260620 09:26:58.306038 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781947562949448-16558-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781947562949448-16558-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781947562949448-16558-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781947562949448-16558-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.43.131:0
--local_ip_for_outbound_sockets=127.16.43.131
--webserver_interface=127.16.43.131
--webserver_port=0
--tserver_master_addrs=127.16.43.190:46261
--builtin_ntp_servers=127.16.43.148:34653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 09:26:58.387955 26627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:26:58.388237 26627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:26:58.388275 26627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:26:58.389497 26627 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:26:58.389578 26627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:26:58.389623 26627 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.43.131
I20260620 09:26:58.391070 26627 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:34653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781947562949448-16558-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781947562949448-16558-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.43.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781947562949448-16558-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.43.131
--webserver_port=0
--tserver_master_addrs=127.16.43.190:46261
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26627
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.43.131
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781947562949448-16558-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:26:58.391358 26627 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:26:58.391827 26627 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:26:58.398618 26632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:26:58.398696 26635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:26:58.398643 26633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:26:58.399114 26627 server_base.cc:1034] running on GCE node
I20260620 09:26:58.399370 26627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:26:58.399686 26627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:26:58.400867 26627 hybrid_clock.cc:648] HybridClock initialized: now 1781947618400842 us; error 38 us; skew 500 ppm
I20260620 09:26:58.402202 26627 webserver.cc:460] Webserver started at http://127.16.43.131:33443/ using document root <none> and password file <none>
I20260620 09:26:58.402456 26627 fs_manager.cc:359] Metadata directory not provided
I20260620 09:26:58.402529 26627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:26:58.402665 26627 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:26:58.403843 26627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781947562949448-16558-0/minicluster-data/ts-2/data/instance:
uuid: "7587826f063d467ca4c77bdd5b75fdbc"
format_stamp: "Formatted at 2026-06-20 09:26:58 on dist-test-slave-706q"
server_key: "0e743f5551026ee71031041e0022b14e"
server_key_iv: "0dc46fb991d1a3fcfde57d7a55b96a27"
server_key_version: "encryptionkey@0"
I20260620 09:26:58.404217 26627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781947562949448-16558-0/minicluster-data/ts-2/wal/instance:
uuid: "7587826f063d467ca4c77bdd5b75fdbc"
format_stamp: "Formatted at 2026-06-20 09:26:58 on dist-test-slave-706q"
server_key: "0e743f5551026ee71031041e0022b14e"
server_key_iv: "0dc46fb991d1a3fcfde57d7a55b96a27"
server_key_version: "encryptionkey@0"
I20260620 09:26:58.405525 26627 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 09:26:58.406306 26641 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:58.406517 26627 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:26:58.406669 26627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781947562949448-16558-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781947562949448-16558-0/minicluster-data/ts-2/wal
uuid: "7587826f063d467ca4c77bdd5b75fdbc"
format_stamp: "Formatted at 2026-06-20 09:26:58 on dist-test-slave-706q"
server_key: "0e743f5551026ee71031041e0022b14e"
server_key_iv: "0dc46fb991d1a3fcfde57d7a55b96a27"
server_key_version: "encryptionkey@0"
I20260620 09:26:58.406764 26627 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781947562949448-16558-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781947562949448-16558-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781947562949448-16558-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:26:58.410558 26627 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:26:58.410877 26627 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:26:58.411036 26627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:26:58.411270 26627 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:26:58.411661 26627 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:26:58.411718 26627 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:58.411759 26627 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:26:58.411789 26627 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:58.417989 26627 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.131:34291
I20260620 09:26:58.418045 26754 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.131:34291 every 8 connection(s)
I20260620 09:26:58.418478 26627 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781947562949448-16558-0/minicluster-data/ts-2/data/info.pb
I20260620 09:26:58.420747 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 26627
I20260620 09:26:58.420852 16558 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781947562949448-16558-0/minicluster-data/ts-2/wal/instance
I20260620 09:26:58.421060 16558 external_mini_cluster.cc:1428] Setting key 245e157f7b2844cd3a1b2e342a089b64
I20260620 09:26:58.421923 26355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:58.419267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.131:57611 (local address 127.16.43.190:46261)
0620 09:26:58.419310 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:58.419312 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:58.419616 (+ 304us) server_negotiation.cc:407] Connection header received
0620 09:26:58.420107 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:58.420108 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:58.420114 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:58.420197 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:58.421594 (+ 1397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:58.421595 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:58.421595 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:26:58.421600 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:58.421618 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:26:58.421623 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:58.421711 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:58.421829 (+ 118us) server_negotiation.cc:299] Negotiation successful
0620 09:26:58.421850 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 09:26:58.422122 26757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:58.419272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:46261 (local address 127.16.43.131:57611)
0620 09:26:58.419448 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:58.419484 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:58.420015 (+ 531us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:58.420256 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:58.420259 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:58.420267 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:58.421461 (+ 1194us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 09:26:58.421461 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:26:58.421493 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 09:26:58.421524 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:26:58.421714 (+ 190us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:26:58.421718 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:26:58.421722 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 09:26:58.421831 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 09:26:58.421898 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":88,"threads_started":1}
I20260620 09:26:58.422559 26755 heartbeater.cc:347] Connected to a master server at 127.16.43.190:46261
I20260620 09:26:58.422677 26755 heartbeater.cc:464] Registering TS with master...
I20260620 09:26:58.422832 26755 heartbeater.cc:511] Master 127.16.43.190:46261 requested a full tablet report, sending...
I20260620 09:26:58.423250 26305 ts_manager.cc:194] Registered new tserver with Master: 7587826f063d467ca4c77bdd5b75fdbc (127.16.43.131:34291)
I20260620 09:26:58.433112 16558 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 09:26:58.435725 26355 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:26:58.434882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44294 (local address 127.16.43.190:46261)
0620 09:26:58.434978 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:58.434981 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:58.435193 (+ 212us) server_negotiation.cc:407] Connection header received
0620 09:26:58.435319 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:58.435320 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:58.435327 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:58.435430 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:58.435634 (+ 204us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44294: BlockingRecv error: recv got EOF from 127.0.0.1:44294 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 09:26:58.435809 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 26365
I20260620 09:26:58.441686 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 26496
I20260620 09:26:58.451643 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 26627
I20260620 09:26:58.457604 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 26274
2026-06-20T09:26:58Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (569 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781947562949448-16558-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:26:58 dist-test-slave-706q krb5kdc[26765](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:26:58 dist-test-slave-706q krb5kdc[26765](info): set up 2 sockets
Jun 20 09:26:58 dist-test-slave-706q krb5kdc[26765](info): commencing operation
krb5kdc: starting...
W20260620 09:26:59.541580 16558 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781947562949448-16558-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781947562949448-16558-0/krb5kdc/test-user.keytab.
Jun 20 09:26:59 dist-test-slave-706q krb5kdc[26765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947619, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:26:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:26:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.43.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.43.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.43.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.190.keytab.
Entry for principal kudu/127.16.43.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.190.keytab.
I20260620 09:26:59.664278 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781947562949448-16558-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781947562949448-16558-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781947562949448-16558-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.43.190:38577
--webserver_interface=127.16.43.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.190.keytab
--principal=kudu/127.16.43.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.43.148:45297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.43.190:38577
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781947562949448-16558-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781947562949448-16558-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781947562949448-16558-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:26:59.747692 26778 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:26:59.747898 26778 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:26:59.747920 26778 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:26:59.748999 26778 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:26:59.749042 26778 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:26:59.749055 26778 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:26:59.749068 26778 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:26:59.749079 26778 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:26:59.750439 26778 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:45297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781947562949448-16558-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781947562949448-16558-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.43.190:38577
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.190.keytab
--principal=kudu/127.16.43.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.43.190:38577
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.43.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26778
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781947562949448-16558-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:26:59.750622 26778 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:26:59.750787 26778 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:26:59.755831 26783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:26:59.755827 26784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:26:59.755827 26786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:26:59.755860 26778 server_base.cc:1034] running on GCE node
I20260620 09:26:59.756218 26778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:26:59.756467 26778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:26:59.757638 26778 hybrid_clock.cc:648] HybridClock initialized: now 1781947619757626 us; error 44 us; skew 500 ppm
Jun 20 09:26:59 dist-test-slave-706q krb5kdc[26765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947619, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:26:59.760300 26778 init.cc:377] Logged in from keytab as kudu/127.16.43.190@KRBTEST.COM (short username kudu)
I20260620 09:26:59.760691 26778 webserver.cc:460] Webserver started at http://127.16.43.190:46383/ using document root <none> and password file <none>
I20260620 09:26:59.760874 26778 fs_manager.cc:359] Metadata directory not provided
I20260620 09:26:59.760919 26778 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:26:59.761006 26778 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:26:59.762058 26778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781947562949448-16558-0/minicluster-data/master-0/data/instance:
uuid: "33c9177f85864fc8bfdfd7f98927d947"
format_stamp: "Formatted at 2026-06-20 09:26:59 on dist-test-slave-706q"
server_key: "54c5d3c42e81dbbb29b49168b784724f"
server_key_iv: "7bf8c5df90e8bd719fc6b2c135db998d"
server_key_version: "encryptionkey@0"
I20260620 09:26:59.762382 26778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781947562949448-16558-0/minicluster-data/master-0/wal/instance:
uuid: "33c9177f85864fc8bfdfd7f98927d947"
format_stamp: "Formatted at 2026-06-20 09:26:59 on dist-test-slave-706q"
server_key: "54c5d3c42e81dbbb29b49168b784724f"
server_key_iv: "7bf8c5df90e8bd719fc6b2c135db998d"
server_key_version: "encryptionkey@0"
I20260620 09:26:59.763767 26778 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 09:26:59.764554 26793 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:59.764735 26778 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:26:59.764842 26778 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781947562949448-16558-0/minicluster-data/master-0/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781947562949448-16558-0/minicluster-data/master-0/wal
uuid: "33c9177f85864fc8bfdfd7f98927d947"
format_stamp: "Formatted at 2026-06-20 09:26:59 on dist-test-slave-706q"
server_key: "54c5d3c42e81dbbb29b49168b784724f"
server_key_iv: "7bf8c5df90e8bd719fc6b2c135db998d"
server_key_version: "encryptionkey@0"
I20260620 09:26:59.764919 26778 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781947562949448-16558-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781947562949448-16558-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781947562949448-16558-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:26:59.778946 26778 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:26:59.781451 26778 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:26:59.781634 26778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:26:59.785630 26778 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.190:38577
I20260620 09:26:59.785684 26845 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.190:38577 every 8 connection(s)
I20260620 09:26:59.786084 26778 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
I20260620 09:26:59.787052 26846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:26:59.789390 26846 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33c9177f85864fc8bfdfd7f98927d947: Bootstrap starting.
I20260620 09:26:59.789628 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 26778
I20260620 09:26:59.789748 16558 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781947562949448-16558-0/minicluster-data/master-0/wal/instance
I20260620 09:26:59.789922 16558 external_mini_cluster.cc:1428] Setting key 7eeff9ee04abf191039ebb429dae5865
I20260620 09:26:59.790078 26846 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 33c9177f85864fc8bfdfd7f98927d947: Neither blocks nor log segments found. Creating new log.
I20260620 09:26:59.790446 26846 log.cc:824] T 00000000000000000000000000000000 P 33c9177f85864fc8bfdfd7f98927d947: Log is configured to *not* fsync() on all Append() calls
I20260620 09:26:59.791157 26846 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33c9177f85864fc8bfdfd7f98927d947: No bootstrap required, opened a new log
I20260620 09:26:59.793745 26846 raft_consensus.cc:348] T 00000000000000000000000000000000 P 33c9177f85864fc8bfdfd7f98927d947 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33c9177f85864fc8bfdfd7f98927d947" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 38577 } }
I20260620 09:26:59.793932 26846 raft_consensus.cc:374] T 00000000000000000000000000000000 P 33c9177f85864fc8bfdfd7f98927d947 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:26:59.793965 26846 raft_consensus.cc:729] T 00000000000000000000000000000000 P 33c9177f85864fc8bfdfd7f98927d947 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33c9177f85864fc8bfdfd7f98927d947, State: Initialized, Role: FOLLOWER
I20260620 09:26:59.794118 26846 consensus_queue.cc:260] T 00000000000000000000000000000000 P 33c9177f85864fc8bfdfd7f98927d947 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33c9177f85864fc8bfdfd7f98927d947" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 38577 } }
I20260620 09:26:59.794211 26846 raft_consensus.cc:388] T 00000000000000000000000000000000 P 33c9177f85864fc8bfdfd7f98927d947 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:26:59.794272 26846 raft_consensus.cc:482] T 00000000000000000000000000000000 P 33c9177f85864fc8bfdfd7f98927d947 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:26:59.794332 26846 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 33c9177f85864fc8bfdfd7f98927d947 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:26:59.794970 26846 raft_consensus.cc:504] T 00000000000000000000000000000000 P 33c9177f85864fc8bfdfd7f98927d947 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33c9177f85864fc8bfdfd7f98927d947" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 38577 } }
I20260620 09:26:59.795085 26846 leader_election.cc:302] T 00000000000000000000000000000000 P 33c9177f85864fc8bfdfd7f98927d947 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 33c9177f85864fc8bfdfd7f98927d947; no voters:
I20260620 09:26:59.795274 26846 leader_election.cc:288] T 00000000000000000000000000000000 P 33c9177f85864fc8bfdfd7f98927d947 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:26:59.795332 26851 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 33c9177f85864fc8bfdfd7f98927d947 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:26:59.795511 26851 raft_consensus.cc:686] T 00000000000000000000000000000000 P 33c9177f85864fc8bfdfd7f98927d947 [term 1 LEADER]: Becoming Leader. State: Replica: 33c9177f85864fc8bfdfd7f98927d947, State: Running, Role: LEADER
I20260620 09:26:59.795578 26846 sys_catalog.cc:564] T 00000000000000000000000000000000 P 33c9177f85864fc8bfdfd7f98927d947 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:26:59.795678 26851 consensus_queue.cc:237] T 00000000000000000000000000000000 P 33c9177f85864fc8bfdfd7f98927d947 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33c9177f85864fc8bfdfd7f98927d947" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 38577 } }
I20260620 09:26:59.796010 26851 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33c9177f85864fc8bfdfd7f98927d947 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 33c9177f85864fc8bfdfd7f98927d947. Latest consensus state: current_term: 1 leader_uuid: "33c9177f85864fc8bfdfd7f98927d947" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33c9177f85864fc8bfdfd7f98927d947" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 38577 } } }
I20260620 09:26:59.796101 26851 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33c9177f85864fc8bfdfd7f98927d947 [sys.catalog]: This master's current role is: LEADER
I20260620 09:26:59.796017 26852 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33c9177f85864fc8bfdfd7f98927d947 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "33c9177f85864fc8bfdfd7f98927d947" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33c9177f85864fc8bfdfd7f98927d947" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 38577 } } }
I20260620 09:26:59.796257 26852 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33c9177f85864fc8bfdfd7f98927d947 [sys.catalog]: This master's current role is: LEADER
I20260620 09:26:59.796406 26858 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:26:59.796918 26858 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 20 09:26:59 dist-test-slave-706q krb5kdc[26765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947619, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.43.190@KRBTEST.COM
I20260620 09:26:59.798566 26858 catalog_manager.cc:1347] Generated new cluster ID: 3908387403e44f3091322f6a5d489977
I20260620 09:26:59.798621 26858 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:26:59.801432 26849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:59.790499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51326 (local address 127.16.43.190:38577)
0620 09:26:59.790695 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:59.790705 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:59.790740 (+ 35us) server_negotiation.cc:407] Connection header received
0620 09:26:59.791223 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:59.791230 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:59.791279 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:59.791457 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:59.792269 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:59.793149 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:59.794243 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:59.794787 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:26:59.794891 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:26:59.797855 (+ 2964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:26:59.797860 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:26:59.797860 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:26:59.797874 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:26:59.799918 (+ 2044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:59.800287 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:59.800290 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:59.800290 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:59.800384 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:26:59.800602 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:26:59.800603 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:26:59.800603 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:26:59.800846 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:26:59.801023 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0620 09:26:59.801095 (+ 72us) server_negotiation.cc:299] Negotiation successful
0620 09:26:59.801263 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":90,"threads_started":1}
I20260620 09:26:59.811316 26858 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:26:59.811942 26858 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:26:59.823463 26858 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 33c9177f85864fc8bfdfd7f98927d947: Generated new TSK 0
I20260620 09:26:59.823766 26858 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.43.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.43.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.43.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.129.keytab.
Entry for principal kudu/127.16.43.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.129.keytab.
I20260620 09:26:59.854451 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781947562949448-16558-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781947562949448-16558-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781947562949448-16558-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781947562949448-16558-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.43.129:0
--local_ip_for_outbound_sockets=127.16.43.129
--webserver_interface=127.16.43.129
--webserver_port=0
--tserver_master_addrs=127.16.43.190:38577
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.129.keytab
--principal=kudu/127.16.43.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.43.148:45297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781947562949448-16558-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781947562949448-16558-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781947562949448-16558-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:26:59.936357 26872 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:26:59.936578 26872 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:26:59.936604 26872 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:26:59.937796 26872 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:26:59.937842 26872 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:26:59.937870 26872 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.43.129
I20260620 09:26:59.939459 26872 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:45297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781947562949448-16558-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781947562949448-16558-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.129.keytab
--principal=kudu/127.16.43.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.43.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781947562949448-16558-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.43.129
--webserver_port=0
--tserver_master_addrs=127.16.43.190:38577
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26872
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.43.129
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781947562949448-16558-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:26:59.939750 26872 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:26:59.940011 26872 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:26:59.945314 26878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:26:59.945314 26880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:26:59.945322 26877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:26:59.945654 26872 server_base.cc:1034] running on GCE node
I20260620 09:26:59.945896 26872 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:26:59.946173 26872 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:26:59.947352 26872 hybrid_clock.cc:648] HybridClock initialized: now 1781947619947329 us; error 44 us; skew 500 ppm
Jun 20 09:26:59 dist-test-slave-706q krb5kdc[26765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947619, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:26:59.950309 26872 init.cc:377] Logged in from keytab as kudu/127.16.43.129@KRBTEST.COM (short username kudu)
I20260620 09:26:59.950762 26872 webserver.cc:460] Webserver started at http://127.16.43.129:35995/ using document root <none> and password file <none>
I20260620 09:26:59.951006 26872 fs_manager.cc:359] Metadata directory not provided
I20260620 09:26:59.951155 26872 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:26:59.951306 26872 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:26:59.952461 26872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781947562949448-16558-0/minicluster-data/ts-0/data/instance:
uuid: "dab59cf7c118457fb84aac584d49148d"
format_stamp: "Formatted at 2026-06-20 09:26:59 on dist-test-slave-706q"
server_key: "c85cf5984627b80ca2bf124187acb5c3"
server_key_iv: "9866ddc2876fb846e3588eae74869e7b"
server_key_version: "encryptionkey@0"
I20260620 09:26:59.952826 26872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781947562949448-16558-0/minicluster-data/ts-0/wal/instance:
uuid: "dab59cf7c118457fb84aac584d49148d"
format_stamp: "Formatted at 2026-06-20 09:26:59 on dist-test-slave-706q"
server_key: "c85cf5984627b80ca2bf124187acb5c3"
server_key_iv: "9866ddc2876fb846e3588eae74869e7b"
server_key_version: "encryptionkey@0"
I20260620 09:26:59.954170 26872 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 09:26:59.954955 26887 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:59.955124 26872 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:59.955250 26872 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781947562949448-16558-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781947562949448-16558-0/minicluster-data/ts-0/wal
uuid: "dab59cf7c118457fb84aac584d49148d"
format_stamp: "Formatted at 2026-06-20 09:26:59 on dist-test-slave-706q"
server_key: "c85cf5984627b80ca2bf124187acb5c3"
server_key_iv: "9866ddc2876fb846e3588eae74869e7b"
server_key_version: "encryptionkey@0"
I20260620 09:26:59.955328 26872 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781947562949448-16558-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781947562949448-16558-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781947562949448-16558-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:26:59.986392 26872 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:26:59.988638 26872 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:26:59.988811 26872 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:26:59.989065 26872 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:26:59.989392 26872 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:26:59.989447 26872 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:59.989492 26872 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:26:59.989521 26872 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:26:59.995522 26872 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.129:41105
I20260620 09:26:59.995617 27000 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.129:41105 every 8 connection(s)
I20260620 09:26:59.995997 26872 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781947562949448-16558-0/minicluster-data/ts-0/data/info.pb
I20260620 09:26:59.999491 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 26872
I20260620 09:26:59.999671 16558 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781947562949448-16558-0/minicluster-data/ts-0/wal/instance
I20260620 09:26:59.999876 16558 external_mini_cluster.cc:1428] Setting key e276dfb26c0d92268895386bad869fe9
Jun 20 09:26:59 dist-test-slave-706q krb5kdc[26765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947619, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.129@KRBTEST.COM for kudu/127.16.43.190@KRBTEST.COM
I20260620 09:27:00.009069 26849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:59.996835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.129:46061 (local address 127.16.43.190:38577)
0620 09:26:59.996904 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0620 09:26:59.996906 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:26:59.997180 (+ 274us) server_negotiation.cc:407] Connection header received
0620 09:26:59.997703 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:26:59.997721 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:26:59.997731 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:26:59.997785 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:26:59.998727 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:26:59.999212 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:27:00.000693 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:27:00.000897 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:27:00.000989 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:27:00.004080 (+ 3091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:27:00.004082 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:27:00.004083 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:27:00.004092 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:27:00.006558 (+ 2466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:27:00.007410 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:27:00.007412 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:27:00.007413 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:27:00.007518 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:27:00.008142 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:27:00.008143 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:27:00.008144 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:27:00.008331 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:27:00.008426 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 09:27:00.008827 (+ 401us) server_negotiation.cc:299] Negotiation successful
0620 09:27:00.008947 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 09:27:00.009146 27003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:26:59.996816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:38577 (local address 127.16.43.129:46061)
0620 09:26:59.996989 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 09:26:59.997034 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 09:26:59.997614 (+ 580us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:26:59.997841 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:26:59.997844 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:26:59.998441 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:26:59.998639 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:26:59.998644 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:26:59.999291 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:26:59.999292 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:27:00.000516 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:27:00.000519 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:27:00.001002 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:27:00.001003 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:27:00.001177 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:27:00.001743 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:27:00.001761 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:27:00.003927 (+ 2166us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:27:00.006801 (+ 2874us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:27:00.006823 (+ 22us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:27:00.006827 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:27:00.007255 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:27:00.007788 (+ 533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:27:00.007791 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:27:00.007791 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:27:00.008019 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:27:00.008462 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:27:00.008472 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:27:00.008653 (+ 181us) client_negotiation.cc:769] Sending connection context
0620 09:27:00.008769 (+ 116us) client_negotiation.cc:240] Negotiation successful
0620 09:27:00.008940 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":83,"threads_started":1}
I20260620 09:27:00.010251 27001 heartbeater.cc:347] Connected to a master server at 127.16.43.190:38577
WARNING: no policy specified for kudu/127.16.43.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.43.130@KRBTEST.COM" created.
I20260620 09:27:00.015642 27001 heartbeater.cc:464] Registering TS with master...
I20260620 09:27:00.015965 27001 heartbeater.cc:511] Master 127.16.43.190:38577 requested a full tablet report, sending...
I20260620 09:27:00.018195 26810 ts_manager.cc:194] Registered new tserver with Master: dab59cf7c118457fb84aac584d49148d (127.16.43.129:41105)
I20260620 09:27:00.019376 26810 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.43.129@KRBTEST.COM'} at 127.16.43.129:46061
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.43.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.130.keytab.
Entry for principal kudu/127.16.43.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.130.keytab.
I20260620 09:27:00.036060 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781947562949448-16558-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781947562949448-16558-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781947562949448-16558-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781947562949448-16558-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.43.130:0
--local_ip_for_outbound_sockets=127.16.43.130
--webserver_interface=127.16.43.130
--webserver_port=0
--tserver_master_addrs=127.16.43.190:38577
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.130.keytab
--principal=kudu/127.16.43.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.43.148:45297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781947562949448-16558-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781947562949448-16558-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781947562949448-16558-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:27:00.119203 27006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:27:00.119473 27006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:27:00.119515 27006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:27:00.120730 27006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:27:00.120811 27006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:27:00.120857 27006 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.43.130
I20260620 09:27:00.122423 27006 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:45297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781947562949448-16558-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781947562949448-16558-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.130.keytab
--principal=kudu/127.16.43.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.43.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781947562949448-16558-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.43.130
--webserver_port=0
--tserver_master_addrs=127.16.43.190:38577
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27006
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.43.130
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781947562949448-16558-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:27:00.122720 27006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:27:00.122977 27006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:27:00.128482 27011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:27:00.128485 27012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:27:00.128482 27014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:27:00.128733 27006 server_base.cc:1034] running on GCE node
I20260620 09:27:00.128973 27006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:27:00.129357 27006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:27:00.130543 27006 hybrid_clock.cc:648] HybridClock initialized: now 1781947620130530 us; error 28 us; skew 500 ppm
Jun 20 09:27:00 dist-test-slave-706q krb5kdc[26765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947620, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:27:00.133363 27006 init.cc:377] Logged in from keytab as kudu/127.16.43.130@KRBTEST.COM (short username kudu)
I20260620 09:27:00.133836 27006 webserver.cc:460] Webserver started at http://127.16.43.130:41481/ using document root <none> and password file <none>
I20260620 09:27:00.134079 27006 fs_manager.cc:359] Metadata directory not provided
I20260620 09:27:00.134126 27006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:27:00.134292 27006 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:27:00.135381 27006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781947562949448-16558-0/minicluster-data/ts-1/data/instance:
uuid: "a2e959fab34943758d6891be1df4ba43"
format_stamp: "Formatted at 2026-06-20 09:27:00 on dist-test-slave-706q"
server_key: "769e26dd6b37b1323c2ecf82caa7e7ac"
server_key_iv: "6bf943a999d61ff6e67d2d87cf155460"
server_key_version: "encryptionkey@0"
I20260620 09:27:00.135792 27006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781947562949448-16558-0/minicluster-data/ts-1/wal/instance:
uuid: "a2e959fab34943758d6891be1df4ba43"
format_stamp: "Formatted at 2026-06-20 09:27:00 on dist-test-slave-706q"
server_key: "769e26dd6b37b1323c2ecf82caa7e7ac"
server_key_iv: "6bf943a999d61ff6e67d2d87cf155460"
server_key_version: "encryptionkey@0"
I20260620 09:27:00.137151 27006 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 09:27:00.138001 27021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:27:00.138243 27006 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 09:27:00.138334 27006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781947562949448-16558-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781947562949448-16558-0/minicluster-data/ts-1/wal
uuid: "a2e959fab34943758d6891be1df4ba43"
format_stamp: "Formatted at 2026-06-20 09:27:00 on dist-test-slave-706q"
server_key: "769e26dd6b37b1323c2ecf82caa7e7ac"
server_key_iv: "6bf943a999d61ff6e67d2d87cf155460"
server_key_version: "encryptionkey@0"
I20260620 09:27:00.138410 27006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781947562949448-16558-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781947562949448-16558-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781947562949448-16558-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:27:00.152472 27006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:27:00.154621 27006 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:27:00.154795 27006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:27:00.155081 27006 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:27:00.155422 27006 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:27:00.155455 27006 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:27:00.155500 27006 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:27:00.155555 27006 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:27:00.161852 27006 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.130:39793
I20260620 09:27:00.161934 27134 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.130:39793 every 8 connection(s)
I20260620 09:27:00.162273 27006 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781947562949448-16558-0/minicluster-data/ts-1/data/info.pb
Jun 20 09:27:00 dist-test-slave-706q krb5kdc[26765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947620, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.130@KRBTEST.COM for kudu/127.16.43.190@KRBTEST.COM
I20260620 09:27:00.171396 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 27006
I20260620 09:27:00.171510 16558 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781947562949448-16558-0/minicluster-data/ts-1/wal/instance
I20260620 09:27:00.171753 16558 external_mini_cluster.cc:1428] Setting key 5cb40cf7411d9b181604e5a8e08dcd86
I20260620 09:27:00.172020 26849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:27:00.163051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.130:43545 (local address 127.16.43.190:38577)
0620 09:27:00.163099 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 09:27:00.163101 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:27:00.163290 (+ 189us) server_negotiation.cc:407] Connection header received
0620 09:27:00.163929 (+ 639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:27:00.163930 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:27:00.163943 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:27:00.163997 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:27:00.165015 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:27:00.165522 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:27:00.166647 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:27:00.166856 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:27:00.166944 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:27:00.169349 (+ 2405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:27:00.169350 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:27:00.169351 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:27:00.169358 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:27:00.170586 (+ 1228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:27:00.170978 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:27:00.170979 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:27:00.170979 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:27:00.171031 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:27:00.171258 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:27:00.171259 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:27:00.171259 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:27:00.171410 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:27:00.171488 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 09:27:00.171856 (+ 368us) server_negotiation.cc:299] Negotiation successful
0620 09:27:00.171949 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 09:27:00.172303 27137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:27:00.162999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:38577 (local address 127.16.43.130:43545)
0620 09:27:00.163147 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 09:27:00.163199 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 09:27:00.163818 (+ 619us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:27:00.164113 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:27:00.164117 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:27:00.164690 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:27:00.164901 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:27:00.164906 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:27:00.165661 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:27:00.165662 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:27:00.166529 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:27:00.166531 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:27:00.166955 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:27:00.166956 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:27:00.167093 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:27:00.167727 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:27:00.167736 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:27:00.169260 (+ 1524us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:27:00.170659 (+ 1399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:27:00.170665 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:27:00.170667 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:27:00.170906 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:27:00.171091 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:27:00.171092 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:27:00.171092 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:27:00.171193 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:27:00.171504 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:27:00.171509 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:27:00.171693 (+ 184us) client_negotiation.cc:769] Sending connection context
0620 09:27:00.171830 (+ 137us) client_negotiation.cc:240] Negotiation successful
0620 09:27:00.172037 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":69,"threads_started":1}
I20260620 09:27:00.172875 27135 heartbeater.cc:347] Connected to a master server at 127.16.43.190:38577
I20260620 09:27:00.172986 27135 heartbeater.cc:464] Registering TS with master...
I20260620 09:27:00.173205 27135 heartbeater.cc:511] Master 127.16.43.190:38577 requested a full tablet report, sending...
I20260620 09:27:00.173669 26810 ts_manager.cc:194] Registered new tserver with Master: a2e959fab34943758d6891be1df4ba43 (127.16.43.130:39793)
I20260620 09:27:00.174412 26810 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.43.130@KRBTEST.COM'} at 127.16.43.130:43545
WARNING: no policy specified for kudu/127.16.43.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.43.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.43.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.131.keytab.
Entry for principal kudu/127.16.43.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.131.keytab.
I20260620 09:27:00.198855 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781947562949448-16558-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781947562949448-16558-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781947562949448-16558-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781947562949448-16558-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.43.131:0
--local_ip_for_outbound_sockets=127.16.43.131
--webserver_interface=127.16.43.131
--webserver_port=0
--tserver_master_addrs=127.16.43.190:38577
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.131.keytab
--principal=kudu/127.16.43.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.43.148:45297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781947562949448-16558-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781947562949448-16558-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781947562949448-16558-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:27:00.282977 27140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:27:00.283211 27140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:27:00.283239 27140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:27:00.284510 27140 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:27:00.284554 27140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:27:00.284582 27140 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.43.131
I20260620 09:27:00.286090 27140 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:45297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781947562949448-16558-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781947562949448-16558-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.131.keytab
--principal=kudu/127.16.43.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.43.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781947562949448-16558-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.43.131
--webserver_port=0
--tserver_master_addrs=127.16.43.190:38577
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27140
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.43.131
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781947562949448-16558-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:27:00.286352 27140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:27:00.286597 27140 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:27:00.291742 27146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:27:00.291749 27145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:27:00.291749 27148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:27:00.292256 27140 server_base.cc:1034] running on GCE node
I20260620 09:27:00.292486 27140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:27:00.292764 27140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:27:00.293937 27140 hybrid_clock.cc:648] HybridClock initialized: now 1781947620293933 us; error 38 us; skew 500 ppm
Jun 20 09:27:00 dist-test-slave-706q krb5kdc[26765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947620, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:27:00.296813 27140 init.cc:377] Logged in from keytab as kudu/127.16.43.131@KRBTEST.COM (short username kudu)
I20260620 09:27:00.297397 27140 webserver.cc:460] Webserver started at http://127.16.43.131:35809/ using document root <none> and password file <none>
I20260620 09:27:00.297636 27140 fs_manager.cc:359] Metadata directory not provided
I20260620 09:27:00.297708 27140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:27:00.297845 27140 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:27:00.298928 27140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781947562949448-16558-0/minicluster-data/ts-2/data/instance:
uuid: "ea30412b0a4440deb0840a555abfe55a"
format_stamp: "Formatted at 2026-06-20 09:27:00 on dist-test-slave-706q"
server_key: "15dfffbf9f5a580a5da30e7af1a46485"
server_key_iv: "16e81f2aa7834852c7ef2967bd7bfa57"
server_key_version: "encryptionkey@0"
I20260620 09:27:00.299319 27140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781947562949448-16558-0/minicluster-data/ts-2/wal/instance:
uuid: "ea30412b0a4440deb0840a555abfe55a"
format_stamp: "Formatted at 2026-06-20 09:27:00 on dist-test-slave-706q"
server_key: "15dfffbf9f5a580a5da30e7af1a46485"
server_key_iv: "16e81f2aa7834852c7ef2967bd7bfa57"
server_key_version: "encryptionkey@0"
I20260620 09:27:00.300761 27140 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:27:00.301542 27155 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:27:00.301718 27140 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 09:27:00.301811 27140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781947562949448-16558-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781947562949448-16558-0/minicluster-data/ts-2/wal
uuid: "ea30412b0a4440deb0840a555abfe55a"
format_stamp: "Formatted at 2026-06-20 09:27:00 on dist-test-slave-706q"
server_key: "15dfffbf9f5a580a5da30e7af1a46485"
server_key_iv: "16e81f2aa7834852c7ef2967bd7bfa57"
server_key_version: "encryptionkey@0"
I20260620 09:27:00.301867 27140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781947562949448-16558-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781947562949448-16558-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781947562949448-16558-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:27:00.324036 27140 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:27:00.326345 27140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:27:00.326459 27140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:27:00.326649 27140 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:27:00.326970 27140 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:27:00.327004 27140 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:27:00.327060 27140 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:27:00.327112 27140 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:27:00.333209 27140 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.131:43547
I20260620 09:27:00.333277 27268 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.131:43547 every 8 connection(s)
I20260620 09:27:00.333629 27140 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781947562949448-16558-0/minicluster-data/ts-2/data/info.pb
I20260620 09:27:00.334126 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 27140
I20260620 09:27:00.334221 16558 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781947562949448-16558-0/minicluster-data/ts-2/wal/instance
I20260620 09:27:00.334424 16558 external_mini_cluster.cc:1428] Setting key 3ff5d595b570722077892450db8e4eaf
Jun 20 09:27:00 dist-test-slave-706q krb5kdc[26765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947620, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.131@KRBTEST.COM for kudu/127.16.43.190@KRBTEST.COM
I20260620 09:27:00.343585 26849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:27:00.334470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.131:41183 (local address 127.16.43.190:38577)
0620 09:27:00.334527 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 09:27:00.334529 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:27:00.334808 (+ 279us) server_negotiation.cc:407] Connection header received
0620 09:27:00.335429 (+ 621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:27:00.335430 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:27:00.335442 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:27:00.335487 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:27:00.336530 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:27:00.337043 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:27:00.338088 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:27:00.338310 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:27:00.338401 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:27:00.340632 (+ 2231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:27:00.340634 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:27:00.340634 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:27:00.340642 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:27:00.342238 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:27:00.342686 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:27:00.342686 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:27:00.342687 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:27:00.342747 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:27:00.342987 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:27:00.342987 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:27:00.342987 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:27:00.343095 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:27:00.343145 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 09:27:00.343402 (+ 257us) server_negotiation.cc:299] Negotiation successful
0620 09:27:00.343464 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 09:27:00.343814 27271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:27:00.334439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:38577 (local address 127.16.43.131:41183)
0620 09:27:00.334594 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 09:27:00.334636 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 09:27:00.335315 (+ 679us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:27:00.335587 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:27:00.335592 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:27:00.336220 (+ 628us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:27:00.336423 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:27:00.336428 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:27:00.337143 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:27:00.337144 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:27:00.337991 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:27:00.337994 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:27:00.338412 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:27:00.338413 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:27:00.338552 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:27:00.339056 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:27:00.339064 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:27:00.340536 (+ 1472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:27:00.342316 (+ 1780us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:27:00.342323 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:27:00.342325 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:27:00.342621 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:27:00.342810 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:27:00.342810 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:27:00.342811 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:27:00.342929 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:27:00.343148 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:27:00.343152 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:27:00.343287 (+ 135us) client_negotiation.cc:769] Sending connection context
0620 09:27:00.343373 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 09:27:00.343497 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":85,"threads_started":1}
I20260620 09:27:00.344477 27269 heartbeater.cc:347] Connected to a master server at 127.16.43.190:38577
I20260620 09:27:00.344587 27269 heartbeater.cc:464] Registering TS with master...
I20260620 09:27:00.344794 27269 heartbeater.cc:511] Master 127.16.43.190:38577 requested a full tablet report, sending...
I20260620 09:27:00.345212 26810 ts_manager.cc:194] Registered new tserver with Master: ea30412b0a4440deb0840a555abfe55a (127.16.43.131:43547)
I20260620 09:27:00.345602 26810 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.43.131@KRBTEST.COM'} at 127.16.43.131:41183
I20260620 09:27:00.346395 16558 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 09:27:00.354156 26849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:27:00.347775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51336 (local address 127.16.43.190:38577)
0620 09:27:00.347859 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0620 09:27:00.347861 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:27:00.347957 (+ 96us) server_negotiation.cc:407] Connection header received
0620 09:27:00.348034 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:27:00.348035 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:27:00.348046 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:27:00.348103 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:27:00.348692 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:27:00.349191 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:27:00.349913 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:27:00.350126 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:27:00.350213 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:27:00.351482 (+ 1269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:27:00.351484 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:27:00.351485 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:27:00.351491 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:27:00.352965 (+ 1474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:27:00.353338 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:27:00.353340 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:27:00.353340 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:27:00.353399 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:27:00.353576 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:27:00.353576 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:27:00.353577 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:27:00.353729 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:27:00.353814 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 09:27:00.353982 (+ 168us) server_negotiation.cc:299] Negotiation successful
0620 09:27:00.354050 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 09:27:00.356261 26810 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51336:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 09:27:00.357899 26810 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 09:27:00.364709 27283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:27:00.360605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59402 (local address 127.16.43.129:41105)
0620 09:27:00.360831 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0620 09:27:00.360836 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:27:00.360846 (+ 10us) server_negotiation.cc:407] Connection header received
0620 09:27:00.360884 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:27:00.360886 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:27:00.360901 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:27:00.361003 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:27:00.361255 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:27:00.362251 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:27:00.363586 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:27:00.364286 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:27:00.364322 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:27:00.364362 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0620 09:27:00.364584 (+ 222us) server_negotiation.cc:299] Negotiation successful
0620 09:27:00.364628 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":118,"threads_started":1}
I20260620 09:27:00.364697 27280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:27:00.360499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.129:41105 (local address 127.0.0.1:59402)
0620 09:27:00.360658 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 09:27:00.360701 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 09:27:00.360768 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:27:00.361005 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:27:00.361008 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:27:00.361011 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:27:00.361087 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:27:00.361089 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:27:00.362362 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:27:00.362363 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:27:00.363467 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:27:00.363469 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:27:00.364410 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:27:00.364411 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:27:00.364465 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:27:00.364475 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 09:27:00.364553 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 09:27:00.364612 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":55,"threads_started":1}
I20260620 09:27:00.365126 27282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:27:00.360515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.131:43547 (local address 127.0.0.1:56712)
0620 09:27:00.361464 (+ 949us) negotiation.cc:107] Waiting for socket to connect
0620 09:27:00.361481 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 09:27:00.361523 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:27:00.361778 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:27:00.361779 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:27:00.361781 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:27:00.361823 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:27:00.361824 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:27:00.363121 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:27:00.363122 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:27:00.364009 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:27:00.364012 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:27:00.364904 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:27:00.364905 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:27:00.364960 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:27:00.364965 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 09:27:00.365020 (+ 55us) client_negotiation.cc:240] Negotiation successful
0620 09:27:00.365056 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":933,"thread_start_us":85,"threads_started":1}
I20260620 09:27:00.365165 27285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:27:00.360736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56712 (local address 127.16.43.131:43547)
0620 09:27:00.360987 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0620 09:27:00.360990 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:27:00.361550 (+ 560us) server_negotiation.cc:407] Connection header received
0620 09:27:00.361601 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:27:00.361603 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:27:00.361619 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:27:00.361699 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:27:00.361917 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:27:00.362997 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:27:00.364123 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:27:00.364818 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:27:00.364898 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:27:00.364954 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 09:27:00.365032 (+ 78us) server_negotiation.cc:299] Negotiation successful
0620 09:27:00.365073 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":93,"threads_started":1}
I20260620 09:27:00.365242 27281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:27:00.360506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.130:39793 (local address 127.0.0.1:42738)
0620 09:27:00.361097 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0620 09:27:00.361107 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 09:27:00.361154 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:27:00.361567 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:27:00.361567 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:27:00.361569 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:27:00.361605 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:27:00.361607 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:27:00.362911 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:27:00.362912 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:27:00.363999 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:27:00.364002 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:27:00.364987 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:27:00.364988 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:27:00.365055 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:27:00.365059 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 09:27:00.365110 (+ 51us) client_negotiation.cc:240] Negotiation successful
0620 09:27:00.365161 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":90,"threads_started":1}
I20260620 09:27:00.365566 27284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:27:00.360744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42738 (local address 127.16.43.130:39793)
0620 09:27:00.360914 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 09:27:00.360919 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:27:00.361217 (+ 298us) server_negotiation.cc:407] Connection header received
0620 09:27:00.361265 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:27:00.361266 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:27:00.361278 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:27:00.361365 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:27:00.361703 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:27:00.362801 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:27:00.364088 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:27:00.364818 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:27:00.364909 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:27:00.364957 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 09:27:00.365446 (+ 489us) server_negotiation.cc:299] Negotiation successful
0620 09:27:00.365491 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":66,"threads_started":1}
I20260620 09:27:00.365820 27203 tablet_service.cc:1467] Processing CreateTablet for tablet d5a0b91f4f124ef6884d321b79729adf (DEFAULT_TABLE table=test-table [id=7bf3f95250cd4b468714508fae1fd61f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:27:00.365850 26935 tablet_service.cc:1467] Processing CreateTablet for tablet d5a0b91f4f124ef6884d321b79729adf (DEFAULT_TABLE table=test-table [id=7bf3f95250cd4b468714508fae1fd61f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:27:00.366196 27203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5a0b91f4f124ef6884d321b79729adf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:27:00.366199 26935 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5a0b91f4f124ef6884d321b79729adf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:27:00.366241 27069 tablet_service.cc:1467] Processing CreateTablet for tablet d5a0b91f4f124ef6884d321b79729adf (DEFAULT_TABLE table=test-table [id=7bf3f95250cd4b468714508fae1fd61f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:27:00.366526 27069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5a0b91f4f124ef6884d321b79729adf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:27:00.369253 27286 tablet_bootstrap.cc:492] T d5a0b91f4f124ef6884d321b79729adf P dab59cf7c118457fb84aac584d49148d: Bootstrap starting.
I20260620 09:27:00.369719 27288 tablet_bootstrap.cc:492] T d5a0b91f4f124ef6884d321b79729adf P a2e959fab34943758d6891be1df4ba43: Bootstrap starting.
I20260620 09:27:00.370034 27286 tablet_bootstrap.cc:654] T d5a0b91f4f124ef6884d321b79729adf P dab59cf7c118457fb84aac584d49148d: Neither blocks nor log segments found. Creating new log.
I20260620 09:27:00.370309 27288 tablet_bootstrap.cc:654] T d5a0b91f4f124ef6884d321b79729adf P a2e959fab34943758d6891be1df4ba43: Neither blocks nor log segments found. Creating new log.
I20260620 09:27:00.370591 27286 log.cc:824] T d5a0b91f4f124ef6884d321b79729adf P dab59cf7c118457fb84aac584d49148d: Log is configured to *not* fsync() on all Append() calls
I20260620 09:27:00.370807 27288 log.cc:824] T d5a0b91f4f124ef6884d321b79729adf P a2e959fab34943758d6891be1df4ba43: Log is configured to *not* fsync() on all Append() calls
I20260620 09:27:00.371122 27287 tablet_bootstrap.cc:492] T d5a0b91f4f124ef6884d321b79729adf P ea30412b0a4440deb0840a555abfe55a: Bootstrap starting.
I20260620 09:27:00.371351 27286 tablet_bootstrap.cc:492] T d5a0b91f4f124ef6884d321b79729adf P dab59cf7c118457fb84aac584d49148d: No bootstrap required, opened a new log
I20260620 09:27:00.371407 27286 ts_tablet_manager.cc:1397] T d5a0b91f4f124ef6884d321b79729adf P dab59cf7c118457fb84aac584d49148d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 09:27:00.371799 27287 tablet_bootstrap.cc:654] T d5a0b91f4f124ef6884d321b79729adf P ea30412b0a4440deb0840a555abfe55a: Neither blocks nor log segments found. Creating new log.
I20260620 09:27:00.372215 27288 tablet_bootstrap.cc:492] T d5a0b91f4f124ef6884d321b79729adf P a2e959fab34943758d6891be1df4ba43: No bootstrap required, opened a new log
I20260620 09:27:00.372323 27287 log.cc:824] T d5a0b91f4f124ef6884d321b79729adf P ea30412b0a4440deb0840a555abfe55a: Log is configured to *not* fsync() on all Append() calls
I20260620 09:27:00.372340 27288 ts_tablet_manager.cc:1397] T d5a0b91f4f124ef6884d321b79729adf P a2e959fab34943758d6891be1df4ba43: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 09:27:00.373162 27287 tablet_bootstrap.cc:492] T d5a0b91f4f124ef6884d321b79729adf P ea30412b0a4440deb0840a555abfe55a: No bootstrap required, opened a new log
I20260620 09:27:00.373239 27287 ts_tablet_manager.cc:1397] T d5a0b91f4f124ef6884d321b79729adf P ea30412b0a4440deb0840a555abfe55a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 09:27:00.373279 27286 raft_consensus.cc:348] T d5a0b91f4f124ef6884d321b79729adf P dab59cf7c118457fb84aac584d49148d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea30412b0a4440deb0840a555abfe55a" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 43547 } } peers { permanent_uuid: "dab59cf7c118457fb84aac584d49148d" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 41105 } } peers { permanent_uuid: "a2e959fab34943758d6891be1df4ba43" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 39793 } }
I20260620 09:27:00.373420 27286 raft_consensus.cc:374] T d5a0b91f4f124ef6884d321b79729adf P dab59cf7c118457fb84aac584d49148d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:27:00.373446 27286 raft_consensus.cc:729] T d5a0b91f4f124ef6884d321b79729adf P dab59cf7c118457fb84aac584d49148d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dab59cf7c118457fb84aac584d49148d, State: Initialized, Role: FOLLOWER
I20260620 09:27:00.373564 27286 consensus_queue.cc:260] T d5a0b91f4f124ef6884d321b79729adf P dab59cf7c118457fb84aac584d49148d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea30412b0a4440deb0840a555abfe55a" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 43547 } } peers { permanent_uuid: "dab59cf7c118457fb84aac584d49148d" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 41105 } } peers { permanent_uuid: "a2e959fab34943758d6891be1df4ba43" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 39793 } }
I20260620 09:27:00.373869 27001 heartbeater.cc:503] Master 127.16.43.190:38577 was elected leader, sending a full tablet report...
I20260620 09:27:00.373855 27286 ts_tablet_manager.cc:1428] T d5a0b91f4f124ef6884d321b79729adf P dab59cf7c118457fb84aac584d49148d: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 09:27:00.374846 27288 raft_consensus.cc:348] T d5a0b91f4f124ef6884d321b79729adf P a2e959fab34943758d6891be1df4ba43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea30412b0a4440deb0840a555abfe55a" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 43547 } } peers { permanent_uuid: "dab59cf7c118457fb84aac584d49148d" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 41105 } } peers { permanent_uuid: "a2e959fab34943758d6891be1df4ba43" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 39793 } }
I20260620 09:27:00.375034 27288 raft_consensus.cc:374] T d5a0b91f4f124ef6884d321b79729adf P a2e959fab34943758d6891be1df4ba43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:27:00.375074 27288 raft_consensus.cc:729] T d5a0b91f4f124ef6884d321b79729adf P a2e959fab34943758d6891be1df4ba43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e959fab34943758d6891be1df4ba43, State: Initialized, Role: FOLLOWER
I20260620 09:27:00.375206 27288 consensus_queue.cc:260] T d5a0b91f4f124ef6884d321b79729adf P a2e959fab34943758d6891be1df4ba43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea30412b0a4440deb0840a555abfe55a" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 43547 } } peers { permanent_uuid: "dab59cf7c118457fb84aac584d49148d" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 41105 } } peers { permanent_uuid: "a2e959fab34943758d6891be1df4ba43" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 39793 } }
I20260620 09:27:00.375240 27287 raft_consensus.cc:348] T d5a0b91f4f124ef6884d321b79729adf P ea30412b0a4440deb0840a555abfe55a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea30412b0a4440deb0840a555abfe55a" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 43547 } } peers { permanent_uuid: "dab59cf7c118457fb84aac584d49148d" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 41105 } } peers { permanent_uuid: "a2e959fab34943758d6891be1df4ba43" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 39793 } }
I20260620 09:27:00.375397 27287 raft_consensus.cc:374] T d5a0b91f4f124ef6884d321b79729adf P ea30412b0a4440deb0840a555abfe55a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:27:00.375438 27287 raft_consensus.cc:729] T d5a0b91f4f124ef6884d321b79729adf P ea30412b0a4440deb0840a555abfe55a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea30412b0a4440deb0840a555abfe55a, State: Initialized, Role: FOLLOWER
I20260620 09:27:00.375494 27288 ts_tablet_manager.cc:1428] T d5a0b91f4f124ef6884d321b79729adf P a2e959fab34943758d6891be1df4ba43: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 09:27:00.375512 27135 heartbeater.cc:503] Master 127.16.43.190:38577 was elected leader, sending a full tablet report...
I20260620 09:27:00.375600 27287 consensus_queue.cc:260] T d5a0b91f4f124ef6884d321b79729adf P ea30412b0a4440deb0840a555abfe55a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea30412b0a4440deb0840a555abfe55a" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 43547 } } peers { permanent_uuid: "dab59cf7c118457fb84aac584d49148d" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 41105 } } peers { permanent_uuid: "a2e959fab34943758d6891be1df4ba43" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 39793 } }
I20260620 09:27:00.375895 27287 ts_tablet_manager.cc:1428] T d5a0b91f4f124ef6884d321b79729adf P ea30412b0a4440deb0840a555abfe55a: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260620 09:27:00.375931 27269 heartbeater.cc:503] Master 127.16.43.190:38577 was elected leader, sending a full tablet report...
W20260620 09:27:00.412859 27136 tablet.cc:2369] T d5a0b91f4f124ef6884d321b79729adf P a2e959fab34943758d6891be1df4ba43: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 09:27:00.462312 27292 raft_consensus.cc:482] T d5a0b91f4f124ef6884d321b79729adf P dab59cf7c118457fb84aac584d49148d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:27:00.462448 27292 raft_consensus.cc:504] T d5a0b91f4f124ef6884d321b79729adf P dab59cf7c118457fb84aac584d49148d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea30412b0a4440deb0840a555abfe55a" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 43547 } } peers { permanent_uuid: "dab59cf7c118457fb84aac584d49148d" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 41105 } } peers { permanent_uuid: "a2e959fab34943758d6891be1df4ba43" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 39793 } }
I20260620 09:27:00.462807 27292 leader_election.cc:288] T d5a0b91f4f124ef6884d321b79729adf P dab59cf7c118457fb84aac584d49148d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea30412b0a4440deb0840a555abfe55a (127.16.43.131:43547), a2e959fab34943758d6891be1df4ba43 (127.16.43.130:39793)
I20260620 09:27:00.467161 27285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:27:00.463148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.129:44349 (local address 127.16.43.131:43547)
0620 09:27:00.463228 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0620 09:27:00.463230 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:27:00.463266 (+ 36us) server_negotiation.cc:407] Connection header received
0620 09:27:00.463302 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:27:00.463304 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:27:00.463315 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:27:00.463390 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:27:00.463805 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:27:00.464510 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:27:00.466157 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:27:00.466769 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:27:00.466845 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:27:00.466855 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 09:27:00.467040 (+ 185us) server_negotiation.cc:299] Negotiation successful
0620 09:27:00.467083 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 09:27:00.467161 27284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:27:00.463148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.129:47735 (local address 127.16.43.130:39793)
0620 09:27:00.463220 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0620 09:27:00.463223 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:27:00.463411 (+ 188us) server_negotiation.cc:407] Connection header received
0620 09:27:00.463459 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:27:00.463460 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:27:00.463468 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:27:00.463583 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:27:00.463757 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:27:00.464541 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:27:00.466157 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:27:00.466759 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:27:00.466846 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:27:00.466855 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 09:27:00.467040 (+ 185us) server_negotiation.cc:299] Negotiation successful
0620 09:27:00.467083 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 09:27:00.467175 27003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:27:00.463049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.131:43547 (local address 127.16.43.129:44349)
0620 09:27:00.463121 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0620 09:27:00.463134 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 09:27:00.463216 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:27:00.463644 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:27:00.463645 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:27:00.463647 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:27:00.463735 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:27:00.463737 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:27:00.464631 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:27:00.464632 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:27:00.466036 (+ 1404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:27:00.466039 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:27:00.466858 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:27:00.466859 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:27:00.466946 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:27:00.466953 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 09:27:00.467016 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 09:27:00.467071 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53,"mutex_wait_us":99}
I20260620 09:27:00.467183 27295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:27:00.463049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.130:39793 (local address 127.16.43.129:47735)
0620 09:27:00.463288 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0620 09:27:00.463299 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 09:27:00.463387 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:27:00.463579 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:27:00.463581 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:27:00.463583 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:27:00.463666 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:27:00.463668 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:27:00.464631 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:27:00.464632 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:27:00.466036 (+ 1404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:27:00.466039 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:27:00.466858 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:27:00.466859 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:27:00.466946 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:27:00.466953 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 09:27:00.467016 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 09:27:00.467070 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"mutex_wait_us":66,"thread_start_us":179,"threads_started":1}
I20260620 09:27:00.467674 27223 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5a0b91f4f124ef6884d321b79729adf" candidate_uuid: "dab59cf7c118457fb84aac584d49148d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea30412b0a4440deb0840a555abfe55a" is_pre_election: true
I20260620 09:27:00.467864 27223 raft_consensus.cc:2454] T d5a0b91f4f124ef6884d321b79729adf P ea30412b0a4440deb0840a555abfe55a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dab59cf7c118457fb84aac584d49148d in term 0.
I20260620 09:27:00.467844 27089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5a0b91f4f124ef6884d321b79729adf" candidate_uuid: "dab59cf7c118457fb84aac584d49148d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e959fab34943758d6891be1df4ba43" is_pre_election: true
I20260620 09:27:00.467979 27089 raft_consensus.cc:2454] T d5a0b91f4f124ef6884d321b79729adf P a2e959fab34943758d6891be1df4ba43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dab59cf7c118457fb84aac584d49148d in term 0.
I20260620 09:27:00.468214 26891 leader_election.cc:302] T d5a0b91f4f124ef6884d321b79729adf P dab59cf7c118457fb84aac584d49148d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dab59cf7c118457fb84aac584d49148d, ea30412b0a4440deb0840a555abfe55a; no voters:
I20260620 09:27:00.468428 27292 raft_consensus.cc:2781] T d5a0b91f4f124ef6884d321b79729adf P dab59cf7c118457fb84aac584d49148d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 09:27:00.468482 27292 raft_consensus.cc:482] T d5a0b91f4f124ef6884d321b79729adf P dab59cf7c118457fb84aac584d49148d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 09:27:00.468501 27292 raft_consensus.cc:3037] T d5a0b91f4f124ef6884d321b79729adf P dab59cf7c118457fb84aac584d49148d [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:27:00.469319 27292 raft_consensus.cc:504] T d5a0b91f4f124ef6884d321b79729adf P dab59cf7c118457fb84aac584d49148d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea30412b0a4440deb0840a555abfe55a" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 43547 } } peers { permanent_uuid: "dab59cf7c118457fb84aac584d49148d" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 41105 } } peers { permanent_uuid: "a2e959fab34943758d6891be1df4ba43" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 39793 } }
I20260620 09:27:00.469482 27292 leader_election.cc:288] T d5a0b91f4f124ef6884d321b79729adf P dab59cf7c118457fb84aac584d49148d [CANDIDATE]: Term 1 election: Requested vote from peers ea30412b0a4440deb0840a555abfe55a (127.16.43.131:43547), a2e959fab34943758d6891be1df4ba43 (127.16.43.130:39793)
I20260620 09:27:00.469673 27089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5a0b91f4f124ef6884d321b79729adf" candidate_uuid: "dab59cf7c118457fb84aac584d49148d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e959fab34943758d6891be1df4ba43"
I20260620 09:27:00.469698 27223 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5a0b91f4f124ef6884d321b79729adf" candidate_uuid: "dab59cf7c118457fb84aac584d49148d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea30412b0a4440deb0840a555abfe55a"
I20260620 09:27:00.469769 27089 raft_consensus.cc:3037] T d5a0b91f4f124ef6884d321b79729adf P a2e959fab34943758d6891be1df4ba43 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:27:00.469766 27223 raft_consensus.cc:3037] T d5a0b91f4f124ef6884d321b79729adf P ea30412b0a4440deb0840a555abfe55a [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:27:00.470504 27089 raft_consensus.cc:2454] T d5a0b91f4f124ef6884d321b79729adf P a2e959fab34943758d6891be1df4ba43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dab59cf7c118457fb84aac584d49148d in term 1.
I20260620 09:27:00.470567 27223 raft_consensus.cc:2454] T d5a0b91f4f124ef6884d321b79729adf P ea30412b0a4440deb0840a555abfe55a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dab59cf7c118457fb84aac584d49148d in term 1.
I20260620 09:27:00.470736 26890 leader_election.cc:302] T d5a0b91f4f124ef6884d321b79729adf P dab59cf7c118457fb84aac584d49148d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2e959fab34943758d6891be1df4ba43, dab59cf7c118457fb84aac584d49148d; no voters:
I20260620 09:27:00.470825 27292 raft_consensus.cc:2781] T d5a0b91f4f124ef6884d321b79729adf P dab59cf7c118457fb84aac584d49148d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:27:00.470991 27292 raft_consensus.cc:686] T d5a0b91f4f124ef6884d321b79729adf P dab59cf7c118457fb84aac584d49148d [term 1 LEADER]: Becoming Leader. State: Replica: dab59cf7c118457fb84aac584d49148d, State: Running, Role: LEADER
I20260620 09:27:00.471108 27292 consensus_queue.cc:237] T d5a0b91f4f124ef6884d321b79729adf P dab59cf7c118457fb84aac584d49148d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea30412b0a4440deb0840a555abfe55a" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 43547 } } peers { permanent_uuid: "dab59cf7c118457fb84aac584d49148d" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 41105 } } peers { permanent_uuid: "a2e959fab34943758d6891be1df4ba43" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 39793 } }
I20260620 09:27:00.472056 26809 catalog_manager.cc:5495] T d5a0b91f4f124ef6884d321b79729adf P dab59cf7c118457fb84aac584d49148d reported cstate change: term changed from 0 to 1, leader changed from <none> to dab59cf7c118457fb84aac584d49148d (127.16.43.129). New cstate: current_term: 1 leader_uuid: "dab59cf7c118457fb84aac584d49148d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea30412b0a4440deb0840a555abfe55a" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 43547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dab59cf7c118457fb84aac584d49148d" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 41105 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2e959fab34943758d6891be1df4ba43" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 39793 } health_report { overall_health: UNKNOWN } } }
I20260620 09:27:00.481784 27283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:27:00.479198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59412 (local address 127.16.43.129:41105)
0620 09:27:00.479268 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0620 09:27:00.479270 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:27:00.479278 (+ 8us) server_negotiation.cc:407] Connection header received
0620 09:27:00.479322 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:27:00.479323 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:27:00.479341 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:27:00.479423 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:27:00.479613 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:27:00.480090 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:27:00.480926 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:27:00.481138 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:27:00.481225 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:27:00.481413 (+ 188us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:27:00.481519 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:27:00.481622 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 09:27:00.481639 (+ 17us) server_negotiation.cc:299] Negotiation successful
0620 09:27:00.481663 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 09:27:00.483773 27223 raft_consensus.cc:1261] T d5a0b91f4f124ef6884d321b79729adf P ea30412b0a4440deb0840a555abfe55a [term 1 FOLLOWER]: Refusing update from remote peer dab59cf7c118457fb84aac584d49148d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:27:00.483814 27089 raft_consensus.cc:1261] T d5a0b91f4f124ef6884d321b79729adf P a2e959fab34943758d6891be1df4ba43 [term 1 FOLLOWER]: Refusing update from remote peer dab59cf7c118457fb84aac584d49148d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:27:00.484200 27296 consensus_queue.cc:1035] T d5a0b91f4f124ef6884d321b79729adf P dab59cf7c118457fb84aac584d49148d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2e959fab34943758d6891be1df4ba43" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 39793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:27:00.484367 27292 consensus_queue.cc:1035] T d5a0b91f4f124ef6884d321b79729adf P dab59cf7c118457fb84aac584d49148d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea30412b0a4440deb0840a555abfe55a" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 43547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:27:00.487426 27299 mvcc.cc:205] Tried to move back new op lower bound from 7298857453498413056 to 7298857453450518528. Current Snapshot: MvccSnapshot[applied={T|T < 7298857453498413056}]
I20260620 09:27:00.490322 26810 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51336:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:27:00.490454 26810 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51336:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:27:00.491267 26810 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 33c9177f85864fc8bfdfd7f98927d947: Sending DeleteTablet for 3 replicas of tablet d5a0b91f4f124ef6884d321b79729adf
I20260620 09:27:00.491653 27203 tablet_service.cc:1514] Processing DeleteTablet for tablet d5a0b91f4f124ef6884d321b79729adf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:27:00 UTC) from {username='kudu', principal='kudu/127.16.43.190@KRBTEST.COM'} at 127.0.0.1:56712
I20260620 09:27:00.491673 27069 tablet_service.cc:1514] Processing DeleteTablet for tablet d5a0b91f4f124ef6884d321b79729adf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:27:00 UTC) from {username='kudu', principal='kudu/127.16.43.190@KRBTEST.COM'} at 127.0.0.1:42738
I20260620 09:27:00.491743 26935 tablet_service.cc:1514] Processing DeleteTablet for tablet d5a0b91f4f124ef6884d321b79729adf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:27:00 UTC) from {username='kudu', principal='kudu/127.16.43.190@KRBTEST.COM'} at 127.0.0.1:59402
I20260620 09:27:00.491961 27311 tablet_replica.cc:330] T d5a0b91f4f124ef6884d321b79729adf P dab59cf7c118457fb84aac584d49148d: stopping tablet replica
I20260620 09:27:00.492061 27311 raft_consensus.cc:2229] T d5a0b91f4f124ef6884d321b79729adf P dab59cf7c118457fb84aac584d49148d [term 1 LEADER]: Raft consensus shutting down.
I20260620 09:27:00.492174 27311 raft_consensus.cc:2258] T d5a0b91f4f124ef6884d321b79729adf P dab59cf7c118457fb84aac584d49148d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:27:00.492267 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 26872
I20260620 09:27:00.492995 27309 tablet_replica.cc:330] T d5a0b91f4f124ef6884d321b79729adf P a2e959fab34943758d6891be1df4ba43: stopping tablet replica
I20260620 09:27:00.493117 27309 raft_consensus.cc:2229] T d5a0b91f4f124ef6884d321b79729adf P a2e959fab34943758d6891be1df4ba43 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:27:00.493247 27309 raft_consensus.cc:2258] T d5a0b91f4f124ef6884d321b79729adf P a2e959fab34943758d6891be1df4ba43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:27:00.493425 27310 tablet_replica.cc:330] T d5a0b91f4f124ef6884d321b79729adf P ea30412b0a4440deb0840a555abfe55a: stopping tablet replica
I20260620 09:27:00.493535 27310 raft_consensus.cc:2229] T d5a0b91f4f124ef6884d321b79729adf P ea30412b0a4440deb0840a555abfe55a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:27:00.493661 27310 raft_consensus.cc:2258] T d5a0b91f4f124ef6884d321b79729adf P ea30412b0a4440deb0840a555abfe55a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:27:00.494174 27309 ts_tablet_manager.cc:1905] T d5a0b91f4f124ef6884d321b79729adf P a2e959fab34943758d6891be1df4ba43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:27:00.494167 27310 ts_tablet_manager.cc:1905] T d5a0b91f4f124ef6884d321b79729adf P ea30412b0a4440deb0840a555abfe55a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:27:00.495896 27309 ts_tablet_manager.cc:1918] T d5a0b91f4f124ef6884d321b79729adf P a2e959fab34943758d6891be1df4ba43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:27:00.495925 27310 ts_tablet_manager.cc:1918] T d5a0b91f4f124ef6884d321b79729adf P ea30412b0a4440deb0840a555abfe55a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:27:00.495985 27310 log.cc:1196] T d5a0b91f4f124ef6884d321b79729adf P ea30412b0a4440deb0840a555abfe55a: Deleting WAL directory at /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781947562949448-16558-0/minicluster-data/ts-2/wal/wals/d5a0b91f4f124ef6884d321b79729adf
I20260620 09:27:00.495985 27309 log.cc:1196] T d5a0b91f4f124ef6884d321b79729adf P a2e959fab34943758d6891be1df4ba43: Deleting WAL directory at /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781947562949448-16558-0/minicluster-data/ts-1/wal/wals/d5a0b91f4f124ef6884d321b79729adf
I20260620 09:27:00.496321 27309 ts_tablet_manager.cc:1939] T d5a0b91f4f124ef6884d321b79729adf P a2e959fab34943758d6891be1df4ba43: Deleting consensus metadata
I20260620 09:27:00.496356 27310 ts_tablet_manager.cc:1939] T d5a0b91f4f124ef6884d321b79729adf P ea30412b0a4440deb0840a555abfe55a: Deleting consensus metadata
I20260620 09:27:00.496783 26797 catalog_manager.cc:4841] TS ea30412b0a4440deb0840a555abfe55a (127.16.43.131:43547): tablet d5a0b91f4f124ef6884d321b79729adf (table test-table [id=7bf3f95250cd4b468714508fae1fd61f]) successfully deleted
I20260620 09:27:00.496847 26796 catalog_manager.cc:4841] TS a2e959fab34943758d6891be1df4ba43 (127.16.43.130:39793): tablet d5a0b91f4f124ef6884d321b79729adf (table test-table [id=7bf3f95250cd4b468714508fae1fd61f]) successfully deleted
W20260620 09:27:00.498800 26794 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.43.129:41105 (error 108)
I20260620 09:27:00.499161 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 27006
I20260620 09:27:00.499861 27280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:27:00.499258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.129:41105 (local address 127.0.0.1:59424)
0620 09:27:00.499770 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0620 09:27:00.499828 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.129:41105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":478}
W20260620 09:27:00.500037 26794 catalog_manager.cc:4584] TS dab59cf7c118457fb84aac584d49148d (127.16.43.129:41105): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d5a0b91f4f124ef6884d321b79729adf: Network error: Client connection negotiation failed: client connection to 127.16.43.129:41105: connect: Connection refused (error 111)
I20260620 09:27:00.504846 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 27140
I20260620 09:27:00.510083 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 26778
2026-06-20T09:27:00Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2053 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781947562949448-16558-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:27:00 dist-test-slave-706q krb5kdc[27315](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:27:00 dist-test-slave-706q krb5kdc[27315](info): set up 2 sockets
Jun 20 09:27:00 dist-test-slave-706q krb5kdc[27315](info): commencing operation
krb5kdc: starting...
W20260620 09:27:01.569620 16558 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781947562949448-16558-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781947562949448-16558-0/krb5kdc/test-user.keytab.
Jun 20 09:27:01 dist-test-slave-706q krb5kdc[27315](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947621, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:27:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:27:01Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.16.43.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.16.43.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.16.43.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781947562949448-16558-0/krb5kdc/rangeradmin_127.16.43.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.16.43.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781947562949448-16558-0/krb5kdc/rangeradmin_127.16.43.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.16.43.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.16.43.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.16.43.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781947562949448-16558-0/krb5kdc/rangerlookup_127.16.43.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.16.43.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781947562949448-16558-0/krb5kdc/rangerlookup_127.16.43.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.16.43.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.16.43.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.16.43.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781947562949448-16558-0/krb5kdc/HTTP_127.16.43.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.16.43.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781947562949448-16558-0/krb5kdc/HTTP_127.16.43.148@KRBTEST.COM.keytab.
I20260620 09:27:01.744364 16558 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781947562949448-16558-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskMkdNwY/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781947562949448-16558-0/minicluster-data/postgres -l logfile start
2026-06-20 09:27:04.418 UTC [27345] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 09:27:04.418 UTC [27345] LOG: listening on IPv4 address "127.16.43.148", port 42037
2026-06-20 09:27:04.421 UTC [27345] LOG: listening on Unix socket "/tmp/.s.PGSQL.42037"
2026-06-20 09:27:04.443 UTC [27348] LOG: database system was shut down at 2026-06-20 09:27:02 UTC
2026-06-20 09:27:04.447 UTC [27345] LOG: database system is ready to accept connections
I20260620 09:27:05.434006 16558 mini_postgres.cc:96] Postgres bound to 42037
2026-06-20 09:27:05.440 UTC [27356] FATAL: database "slave" does not exist
127.16.43.148:42037 - accepting connections
I20260620 09:27:05.441182 16558 mini_ranger.cc:161] Starting Ranger...
I20260620 09:27:05.456132 16558 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 09:27:05.969033 16558 mini_ranger.cc:87] Created ranger Postgres database
I20260620 09:27:05.969105 16558 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781947562949448-16558-0/minicluster-data/ranger-admin
2026-06-20 09:27:06,251 [I] DB FLAVOR :POSTGRES
2026-06-20 09:27:06,251 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 09:27:06,251 [I] Checking connection..
2026-06-20 09:27:06,252 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMkdNwY/build/release/bin/postgresql.jar:/tmp/dist-test-taskMkdNwY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.43.148:42037/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 09:27:06,518 [I] Checking connection passed.
2026-06-20 09:27:06,518 [I] --------- Verifying version history table ---------
2026-06-20 09:27:06,518 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMkdNwY/build/release/bin/postgresql.jar:/tmp/dist-test-taskMkdNwY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.43.148:42037/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 09:27:06,787 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 09:27:06,788 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMkdNwY/build/release/bin/postgresql.jar:/tmp/dist-test-taskMkdNwY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.43.148:42037/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 09:27:07,023 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 09:27:07,023 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 09:27:07,023 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMkdNwY/build/release/bin/postgresql.jar:/tmp/dist-test-taskMkdNwY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.43.148:42037/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskMkdNwY/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 09:27:07.281 UTC [27420] WARNING: there is no transaction in progress
2026-06-20 09:27:07,287 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 09:27:07,288 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMkdNwY/build/release/bin/postgresql.jar:/tmp/dist-test-taskMkdNwY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.43.148:42037/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 09:27:07,546 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 09:27:07,546 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 09:27:07,546 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMkdNwY/build/release/bin/postgresql.jar:/tmp/dist-test-taskMkdNwY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.43.148:42037/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 09:27:07,807 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMkdNwY/build/release/bin/postgresql.jar:/tmp/dist-test-taskMkdNwY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.43.148:42037/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 09:27:08,053 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMkdNwY/build/release/bin/postgresql.jar:/tmp/dist-test-taskMkdNwY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.43.148:42037/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-706q.c.gcp-upstream.internal','N') ;"
2026-06-20 09:27:08,285 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMkdNwY/build/release/bin/postgresql.jar:/tmp/dist-test-taskMkdNwY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.43.148:42037/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 09:27:08,531 [I] Table x_portal_user does not exist in database ranger
2026-06-20 09:27:08,531 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMkdNwY/build/release/bin/postgresql.jar:/tmp/dist-test-taskMkdNwY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.43.148:42037/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 09:27:08,786 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 09:27:08,787 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMkdNwY/build/release/bin/postgresql.jar:/tmp/dist-test-taskMkdNwY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.43.148:42037/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 09:27:09,028 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMkdNwY/build/release/bin/postgresql.jar:/tmp/dist-test-taskMkdNwY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.43.148:42037/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 09:27:09,277 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 09:27:09,278 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMkdNwY/build/release/bin/postgresql.jar:/tmp/dist-test-taskMkdNwY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.43.148:42037/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskMkdNwY/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 09:27:10.293 UTC [27528] WARNING: there is no transaction in progress
2026-06-20 09:27:10.310 UTC [27528] WARNING: there is no transaction in progress
2026-06-20 09:27:10.327 UTC [27528] WARNING: there is no transaction in progress
2026-06-20 09:27:10.343 UTC [27528] WARNING: there is no transaction in progress
2026-06-20 09:27:10.363 UTC [27528] WARNING: there is no transaction in progress
2026-06-20 09:27:10.384 UTC [27528] WARNING: there is no transaction in progress
2026-06-20 09:27:10.454 UTC [27528] WARNING: there is no transaction in progress
2026-06-20 09:27:10.462 UTC [27528] WARNING: there is no transaction in progress
2026-06-20 09:27:10.477 UTC [27528] WARNING: there is no transaction in progress
2026-06-20 09:27:10.487 UTC [27528] WARNING: there is no transaction in progress
2026-06-20 09:27:10.497 UTC [27528] WARNING: there is no transaction in progress
2026-06-20 09:27:10.511 UTC [27528] WARNING: there is no transaction in progress
2026-06-20 09:27:10.522 UTC [27528] WARNING: there is no transaction in progress
2026-06-20 09:27:10.528 UTC [27528] WARNING: there is no transaction in progress
2026-06-20 09:27:10.536 UTC [27528] WARNING: there is no transaction in progress
2026-06-20 09:27:10.544 UTC [27528] WARNING: there is no transaction in progress
2026-06-20 09:27:11,217 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 09:27:11,218 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMkdNwY/build/release/bin/postgresql.jar:/tmp/dist-test-taskMkdNwY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.43.148:42037/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 09:27:11,476 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 09:27:11,477 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMkdNwY/build/release/bin/postgresql.jar:/tmp/dist-test-taskMkdNwY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.43.148:42037/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 09:27:11,753 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 09:27:11,753 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMkdNwY/build/release/bin/postgresql.jar:/tmp/dist-test-taskMkdNwY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.43.148:42037/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 09:27:12,020 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 09:27:12,020 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMkdNwY/build/release/bin/postgresql.jar:/tmp/dist-test-taskMkdNwY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.43.148:42037/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 09:27:12,272 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMkdNwY/build/release/bin/postgresql.jar:/tmp/dist-test-taskMkdNwY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.43.148:42037/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 09:27:12,546 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMkdNwY/build/release/bin/postgresql.jar:/tmp/dist-test-taskMkdNwY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.43.148:42037/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-706q.c.gcp-upstream.internal';"
2026-06-20 09:27:12,802 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 09:27:12,802 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMkdNwY/build/release/bin/postgresql.jar:/tmp/dist-test-taskMkdNwY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.43.148:42037/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 09:27:13,044 [I] DB_PATCHES have already been applied
I20260620 09:27:13.049661 16558 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781947562949448-16558-0/minicluster-data/ranger-admin:/tmp/dist-test-taskMkdNwY/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskMkdNwY/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskMkdNwY/build/release/bin/postgresql.jar:/tmp/dist-test-taskMkdNwY/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 09:27:13.049746 16558 mini_ranger.cc:193] Using host: 127.16.43.148
I20260620 09:27:13.053058 16558 mini_ranger.cc:227] Ranger admin URL: http://127.16.43.148:55261
Jun 20, 2026 9:27:13 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 9:27:13 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 9:27:13 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781947562949448-16558-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 9:27:13 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781947562949448-16558-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 9:27:13 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781947562949448-16558-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 9:27:13 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781947562949448-16558-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 9:27:13 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 09:27:14.195590 16558 mini_ranger.cc:160] Time spent starting Ranger: real 8.754s user 0.001s sys 0.009s
Jun 20, 2026 9:27:35 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 9:27:35 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 20, 2026 9:27:35 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 9:27:35 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 9:27:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 9:27:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 9:27:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 9:27:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 9:27:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 9:27:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 9:27:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 9:27:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 9:27:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 9:27:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 9:27:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 9:27:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 9:27:35 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 9:27:36 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 09:27:36.641558 16558 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.16.43.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.16.43.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.16.43.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781947562949448-16558-0/krb5kdc/rangerkms_127.16.43.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.16.43.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781947562949448-16558-0/krb5kdc/rangerkms_127.16.43.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.16.43.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.16.43.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.16.43.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781947562949448-16558-0/krb5kdc/HTTP_127.16.43.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.16.43.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781947562949448-16558-0/krb5kdc/HTTP_127.16.43.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 09:27:36 dist-test-slave-706q krb5kdc[27315](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947656, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 09:27:36.742470 16558 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 09:27:36.753050 16558 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 09:27:37.710052 16558 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 09:27:37.710140 16558 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781947562949448-16558-0/minicluster-data/ranger-kms
I20260620 09:27:37.710163 16558 mini_ranger_kms.cc:227] Using postgres at 127.16.43.148:42037
2026-06-20 09:27:37,881 [I] DB FLAVOR :POSTGRES
2026-06-20 09:27:37,881 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 09:27:37,881 [I] Checking connection
2026-06-20 09:27:37,881 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMkdNwY/build/release/bin/postgresql.jar:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781947562949448-16558-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.43.148:42037/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 09:27:38,137 [I] connection success
2026-06-20 09:27:38,137 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 09:27:38,137 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 09:27:38,137 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMkdNwY/build/release/bin/postgresql.jar:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781947562949448-16558-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.43.148:42037/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 09:27:38,375 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 09:27:38,376 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 09:27:38,376 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 09:27:38,376 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMkdNwY/build/release/bin/postgresql.jar:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781947562949448-16558-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.43.148:42037/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781947562949448-16558-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 09:27:38,646 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 09:27:39.113632 16558 mini_ranger_kms.cc:310] Created kms service
I20260620 09:27:39.294303 16558 mini_ranger_kms.cc:326] Created kudu user
I20260620 09:27:39.342808 16558 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 09:27:39 dist-test-slave-706q krb5kdc[27315](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947656, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.16.43.148@KRBTEST.COM
I20260620 09:27:39.738555 16558 mini_ranger_kms.cc:384] Added ranger policy
I20260620 09:27:39.738677 16558 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781947562949448-16558-0/minicluster-data/ranger-kms:/tmp/dist-test-taskMkdNwY/build/release/bin/postgresql.jar:/tmp/dist-test-taskMkdNwY/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskMkdNwY/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 09:27:39.738703 16558 mini_ranger_kms.cc:240] Using host: 127.16.43.148
I20260620 09:27:39.743891 16558 mini_ranger_kms.cc:276] Ranger KMS PID: 27708
I20260620 09:27:39.744016 16558 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.16.43.148:53685
Jun 20, 2026 9:27:40 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 9:27:40 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 9:27:40 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781947562949448-16558-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 9:27:40 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781947562949448-16558-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 9:27:41 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781947562949448-16558-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 9:27:41 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260620 09:27:42.083954 16558 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.341s user 0.002s sys 0.152s
I20260620 09:27:42.084059 16558 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 09:27:42.084132 16558 mini_ranger_kms.cc:401] 127.16.43.148:53685/kms/v1/keys
Jun 20 09:27:42 dist-test-slave-706q krb5kdc[27315](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781947662, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.16.43.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 09:27:42 dist-test-slave-706q krb5kdc[27315](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947662, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.16.43.148@KRBTEST.COM for HTTP/127.16.43.148@KRBTEST.COM
Jun 20 09:27:43 dist-test-slave-706q krb5kdc[27315](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947662, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.16.43.148@KRBTEST.COM for HTTP/127.16.43.148@KRBTEST.COM
Jun 20 09:27:45 dist-test-slave-706q krb5kdc[27315](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947665, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.16.43.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.43.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.43.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.190.keytab.
Entry for principal kudu/127.16.43.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.190.keytab.
I20260620 09:27:45.458760 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781947562949448-16558-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781947562949448-16558-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781947562949448-16558-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.43.190:34201
--webserver_interface=127.16.43.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.190.keytab
--principal=kudu/127.16.43.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.43.148:42679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.43.190:34201
--ranger_config_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781947562949448-16558-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.43.148:53685/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781947562949448-16558-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781947562949448-16558-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781947562949448-16558-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:27:45.542513 27759 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:27:45.542732 27759 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:27:45.542754 27759 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:27:45.544056 27759 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:27:45.544127 27759 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:27:45.544143 27759 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:27:45.544155 27759 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:27:45.544167 27759 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:27:45.545763 27759 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:42679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781947562949448-16558-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781947562949448-16558-0/minicluster-data/master-0/wal
--ranger_kms_url=127.16.43.148:53685/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.16.43.190:34201
--ranger_config_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781947562949448-16558-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.190.keytab
--principal=kudu/127.16.43.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.43.190:34201
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.43.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27759
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781947562949448-16558-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:27:45.546027 27759 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:27:45.546285 27759 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:27:45.551784 27764 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:27:45.551841 27759 server_base.cc:1034] running on GCE node
W20260620 09:27:45.551939 27767 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:27:45.551802 27765 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:27:45.552297 27759 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:27:45.552654 27759 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:27:45.553833 27759 hybrid_clock.cc:648] HybridClock initialized: now 1781947665553820 us; error 33 us; skew 500 ppm
Jun 20 09:27:45 dist-test-slave-706q krb5kdc[27315](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947665, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:27:45.556906 27759 init.cc:377] Logged in from keytab as kudu/127.16.43.190@KRBTEST.COM (short username kudu)
I20260620 09:27:45.557402 27759 webserver.cc:460] Webserver started at http://127.16.43.190:45233/ using document root <none> and password file <none>
I20260620 09:27:45.557636 27759 fs_manager.cc:359] Metadata directory not provided
I20260620 09:27:45.557699 27759 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:27:45.557866 27759 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 09:27:45 dist-test-slave-706q krb5kdc[27315](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947665, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.190@KRBTEST.COM for HTTP/127.16.43.148@KRBTEST.COM
I20260620 09:27:45.610972 27759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781947562949448-16558-0/minicluster-data/master-0/data/instance:
uuid: "be5fa4a54bc9438789b0b97171758dc6"
format_stamp: "Formatted at 2026-06-20 09:27:45 on dist-test-slave-706q"
server_key: "27c61de8c6ac97be0ef5a105640d2f83"
server_key_iv: "eff4bb425340fd5b8bb4aad6b10a9bfd"
server_key_version: "kuduclusterkey@0"
I20260620 09:27:45.611405 27759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781947562949448-16558-0/minicluster-data/master-0/wal/instance:
uuid: "be5fa4a54bc9438789b0b97171758dc6"
format_stamp: "Formatted at 2026-06-20 09:27:45 on dist-test-slave-706q"
server_key: "27c61de8c6ac97be0ef5a105640d2f83"
server_key_iv: "eff4bb425340fd5b8bb4aad6b10a9bfd"
server_key_version: "kuduclusterkey@0"
I20260620 09:27:45.612927 27759 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 09:27:45.625626 27774 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:27:45.625893 27759 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 09:27:45.626053 27759 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781947562949448-16558-0/minicluster-data/master-0/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781947562949448-16558-0/minicluster-data/master-0/wal
uuid: "be5fa4a54bc9438789b0b97171758dc6"
format_stamp: "Formatted at 2026-06-20 09:27:45 on dist-test-slave-706q"
server_key: "27c61de8c6ac97be0ef5a105640d2f83"
server_key_iv: "eff4bb425340fd5b8bb4aad6b10a9bfd"
server_key_version: "kuduclusterkey@0"
I20260620 09:27:45.626263 27759 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781947562949448-16558-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781947562949448-16558-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781947562949448-16558-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:27:45.652320 27759 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:27:45.654487 27759 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:27:45.654663 27759 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:27:45.659621 27759 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.190:34201
I20260620 09:27:45.659646 27826 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.190:34201 every 8 connection(s)
I20260620 09:27:45.660079 27759 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
I20260620 09:27:45.660778 27827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:27:45.663036 27827 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be5fa4a54bc9438789b0b97171758dc6: Bootstrap starting.
I20260620 09:27:45.663700 27827 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P be5fa4a54bc9438789b0b97171758dc6: Neither blocks nor log segments found. Creating new log.
I20260620 09:27:45.663996 27827 log.cc:824] T 00000000000000000000000000000000 P be5fa4a54bc9438789b0b97171758dc6: Log is configured to *not* fsync() on all Append() calls
I20260620 09:27:45.664680 27827 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be5fa4a54bc9438789b0b97171758dc6: No bootstrap required, opened a new log
I20260620 09:27:45.665434 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 27759
I20260620 09:27:45.665539 16558 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781947562949448-16558-0/minicluster-data/master-0/wal/instance
I20260620 09:27:45.665733 16558 external_mini_cluster.cc:1428] Setting key 0dec37c2ec86bd9424df8b2f4e2705a9
I20260620 09:27:45.667249 27827 raft_consensus.cc:348] T 00000000000000000000000000000000 P be5fa4a54bc9438789b0b97171758dc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be5fa4a54bc9438789b0b97171758dc6" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 34201 } }
I20260620 09:27:45.667425 27827 raft_consensus.cc:374] T 00000000000000000000000000000000 P be5fa4a54bc9438789b0b97171758dc6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:27:45.667487 27827 raft_consensus.cc:729] T 00000000000000000000000000000000 P be5fa4a54bc9438789b0b97171758dc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be5fa4a54bc9438789b0b97171758dc6, State: Initialized, Role: FOLLOWER
I20260620 09:27:45.667689 27827 consensus_queue.cc:260] T 00000000000000000000000000000000 P be5fa4a54bc9438789b0b97171758dc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be5fa4a54bc9438789b0b97171758dc6" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 34201 } }
I20260620 09:27:45.667794 27827 raft_consensus.cc:388] T 00000000000000000000000000000000 P be5fa4a54bc9438789b0b97171758dc6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:27:45.667830 27827 raft_consensus.cc:482] T 00000000000000000000000000000000 P be5fa4a54bc9438789b0b97171758dc6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:27:45.667910 27827 raft_consensus.cc:3037] T 00000000000000000000000000000000 P be5fa4a54bc9438789b0b97171758dc6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:27:45.668664 27827 raft_consensus.cc:504] T 00000000000000000000000000000000 P be5fa4a54bc9438789b0b97171758dc6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be5fa4a54bc9438789b0b97171758dc6" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 34201 } }
I20260620 09:27:45.668792 27827 leader_election.cc:302] T 00000000000000000000000000000000 P be5fa4a54bc9438789b0b97171758dc6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: be5fa4a54bc9438789b0b97171758dc6; no voters:
I20260620 09:27:45.669011 27827 leader_election.cc:288] T 00000000000000000000000000000000 P be5fa4a54bc9438789b0b97171758dc6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:27:45.669039 27832 raft_consensus.cc:2781] T 00000000000000000000000000000000 P be5fa4a54bc9438789b0b97171758dc6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:27:45.669261 27832 raft_consensus.cc:686] T 00000000000000000000000000000000 P be5fa4a54bc9438789b0b97171758dc6 [term 1 LEADER]: Becoming Leader. State: Replica: be5fa4a54bc9438789b0b97171758dc6, State: Running, Role: LEADER
I20260620 09:27:45.669332 27827 sys_catalog.cc:564] T 00000000000000000000000000000000 P be5fa4a54bc9438789b0b97171758dc6 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:27:45.669417 27832 consensus_queue.cc:237] T 00000000000000000000000000000000 P be5fa4a54bc9438789b0b97171758dc6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be5fa4a54bc9438789b0b97171758dc6" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 34201 } }
I20260620 09:27:45.669818 27832 sys_catalog.cc:455] T 00000000000000000000000000000000 P be5fa4a54bc9438789b0b97171758dc6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "be5fa4a54bc9438789b0b97171758dc6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be5fa4a54bc9438789b0b97171758dc6" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 34201 } } }
I20260620 09:27:45.670028 27833 sys_catalog.cc:455] T 00000000000000000000000000000000 P be5fa4a54bc9438789b0b97171758dc6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader be5fa4a54bc9438789b0b97171758dc6. Latest consensus state: current_term: 1 leader_uuid: "be5fa4a54bc9438789b0b97171758dc6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be5fa4a54bc9438789b0b97171758dc6" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 34201 } } }
I20260620 09:27:45.670926 27832 sys_catalog.cc:458] T 00000000000000000000000000000000 P be5fa4a54bc9438789b0b97171758dc6 [sys.catalog]: This master's current role is: LEADER
I20260620 09:27:45.670974 27833 sys_catalog.cc:458] T 00000000000000000000000000000000 P be5fa4a54bc9438789b0b97171758dc6 [sys.catalog]: This master's current role is: LEADER
I20260620 09:27:45.671131 27846 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:27:45.671653 27846 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:27:45.670804 27827 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781947562949448-16558-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
Jun 20 09:27:45 dist-test-slave-706q krb5kdc[27315](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947665, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.43.190@KRBTEST.COM
I20260620 09:27:45.677374 27846 catalog_manager.cc:1347] Generated new cluster ID: 3f30df46cc0141a59ed9d8812cea2cfc
I20260620 09:27:45.677510 27846 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:27:45.680198 27831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:27:45.666280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55490 (local address 127.16.43.190:34201)
0620 09:27:45.666483 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 09:27:45.666493 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 09:27:45.666525 (+ 32us) server_negotiation.cc:407] Connection header received
0620 09:27:45.667259 (+ 734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:27:45.667270 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:27:45.667348 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:27:45.667506 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:27:45.668427 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:27:45.669874 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:27:45.671063 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:27:45.671713 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:27:45.671839 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:27:45.674775 (+ 2936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:27:45.674823 (+ 48us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:27:45.674823 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:27:45.674958 (+ 135us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:27:45.678103 (+ 3145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:27:45.678514 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:27:45.678518 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:27:45.678518 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:27:45.678620 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:27:45.678855 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:27:45.678856 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:27:45.678856 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:27:45.679162 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:27:45.679350 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0620 09:27:45.679421 (+ 71us) server_negotiation.cc:299] Negotiation successful
0620 09:27:45.679748 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":97,"threads_started":1}
I20260620 09:27:45.693746 27846 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:27:45.694525 27846 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:27:45.707638 27846 catalog_manager.cc:5868] T 00000000000000000000000000000000 P be5fa4a54bc9438789b0b97171758dc6: Generated new TSK 0
I20260620 09:27:45.707873 27846 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.43.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.43.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.43.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.129.keytab.
Entry for principal kudu/127.16.43.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.129.keytab.
I20260620 09:27:46.399075 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781947562949448-16558-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781947562949448-16558-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781947562949448-16558-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781947562949448-16558-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.43.148:53685/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.43.129:0
--local_ip_for_outbound_sockets=127.16.43.129
--webserver_interface=127.16.43.129
--webserver_port=0
--tserver_master_addrs=127.16.43.190:34201
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.129.keytab
--principal=kudu/127.16.43.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.43.148:42679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781947562949448-16558-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781947562949448-16558-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781947562949448-16558-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:27:46.504791 27872 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:27:46.504987 27872 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:27:46.505012 27872 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:27:46.506352 27872 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:27:46.506430 27872 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:27:46.506470 27872 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.43.129
I20260620 09:27:46.508661 27872 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:42679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781947562949448-16558-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781947562949448-16558-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.16.43.148:53685/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.129.keytab
--principal=kudu/127.16.43.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.43.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781947562949448-16558-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.43.129
--webserver_port=0
--tserver_master_addrs=127.16.43.190:34201
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27872
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.43.129
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781947562949448-16558-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:27:46.508920 27872 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:27:46.509222 27872 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:27:46.515095 27877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:27:46.515096 27878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:27:46.515396 27880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:27:46.515782 27872 server_base.cc:1034] running on GCE node
I20260620 09:27:46.515945 27872 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:27:46.516147 27872 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:27:46.517314 27872 hybrid_clock.cc:648] HybridClock initialized: now 1781947666517301 us; error 34 us; skew 500 ppm
Jun 20 09:27:46 dist-test-slave-706q krb5kdc[27315](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947666, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:27:46.523830 27872 init.cc:377] Logged in from keytab as kudu/127.16.43.129@KRBTEST.COM (short username kudu)
I20260620 09:27:46.524411 27872 webserver.cc:460] Webserver started at http://127.16.43.129:44225/ using document root <none> and password file <none>
I20260620 09:27:46.524693 27872 fs_manager.cc:359] Metadata directory not provided
I20260620 09:27:46.524755 27872 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:27:46.524946 27872 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 09:27:46 dist-test-slave-706q krb5kdc[27315](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947666, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.129@KRBTEST.COM for HTTP/127.16.43.148@KRBTEST.COM
I20260620 09:27:46.540225 27872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781947562949448-16558-0/minicluster-data/ts-0/data/instance:
uuid: "d654c4732f754aca8098d35c4e1ae4bd"
format_stamp: "Formatted at 2026-06-20 09:27:46 on dist-test-slave-706q"
server_key: "67cec78df690b6ef6244656507c79676"
server_key_iv: "de00001700ead4f891379bd29dcd75b1"
server_key_version: "kuduclusterkey@0"
I20260620 09:27:46.540776 27872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781947562949448-16558-0/minicluster-data/ts-0/wal/instance:
uuid: "d654c4732f754aca8098d35c4e1ae4bd"
format_stamp: "Formatted at 2026-06-20 09:27:46 on dist-test-slave-706q"
server_key: "67cec78df690b6ef6244656507c79676"
server_key_iv: "de00001700ead4f891379bd29dcd75b1"
server_key_version: "kuduclusterkey@0"
I20260620 09:27:46.542889 27872 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260620 09:27:46.559213 27889 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:27:46.559465 27872 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.002s
I20260620 09:27:46.559612 27872 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781947562949448-16558-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781947562949448-16558-0/minicluster-data/ts-0/wal
uuid: "d654c4732f754aca8098d35c4e1ae4bd"
format_stamp: "Formatted at 2026-06-20 09:27:46 on dist-test-slave-706q"
server_key: "67cec78df690b6ef6244656507c79676"
server_key_iv: "de00001700ead4f891379bd29dcd75b1"
server_key_version: "kuduclusterkey@0"
I20260620 09:27:46.559825 27872 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781947562949448-16558-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781947562949448-16558-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781947562949448-16558-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:27:46.572206 27872 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:27:46.574572 27872 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:27:46.574795 27872 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:27:46.575120 27872 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:27:46.575752 27872 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:27:46.575855 27872 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:27:46.575973 27872 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:27:46.576058 27872 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:27:46.582953 27872 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.129:46487
I20260620 09:27:46.583495 27872 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781947562949448-16558-0/minicluster-data/ts-0/data/info.pb
I20260620 09:27:46.589884 28002 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.129:46487 every 8 connection(s)
I20260620 09:27:46.591790 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 27872
I20260620 09:27:46.591917 16558 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781947562949448-16558-0/minicluster-data/ts-0/wal/instance
I20260620 09:27:46.592146 16558 external_mini_cluster.cc:1428] Setting key 4de4eda7dcba9cc5486e4f4f2dedbc5c
Jun 20 09:27:46 dist-test-slave-706q krb5kdc[27315](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947666, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.129@KRBTEST.COM for kudu/127.16.43.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.43.130@KRBTEST.COM; defaulting to no policy
I20260620 09:27:46.610114 28005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:27:46.590598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:34201 (local address 127.16.43.129:56719)
0620 09:27:46.591679 (+ 1081us) negotiation.cc:107] Waiting for socket to connect
0620 09:27:46.591731 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 09:27:46.593089 (+ 1358us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:27:46.599993 (+ 6904us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:27:46.599999 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:27:46.600306 (+ 307us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:27:46.600663 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:27:46.600672 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:27:46.601748 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:27:46.601749 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:27:46.603111 (+ 1362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:27:46.603114 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:27:46.603704 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:27:46.603705 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:27:46.603853 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:27:46.604518 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:27:46.604529 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:27:46.606249 (+ 1720us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:27:46.608434 (+ 2185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:27:46.608441 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:27:46.608444 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:27:46.608732 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:27:46.608968 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:27:46.608969 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:27:46.608969 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:27:46.609107 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:27:46.609415 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:27:46.609422 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:27:46.609590 (+ 168us) client_negotiation.cc:769] Sending connection context
0620 09:27:46.609699 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 09:27:46.609857 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1034,"thread_start_us":107,"threads_started":1}
I20260620 09:27:46.610908 28006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:27:46.590937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.129:56719 (local address 127.16.43.190:34201)
0620 09:27:46.599631 (+ 8694us) server_negotiation.cc:206] Beginning negotiation
0620 09:27:46.599633 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:27:46.599648 (+ 15us) server_negotiation.cc:407] Connection header received
0620 09:27:46.599678 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:27:46.599679 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:27:46.599695 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:27:46.599795 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:27:46.600801 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:27:46.601624 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:27:46.603215 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:27:46.603559 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:27:46.603628 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:27:46.606369 (+ 2741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:27:46.606391 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:27:46.606391 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:27:46.606399 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:27:46.608329 (+ 1930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:27:46.608818 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:27:46.608819 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:27:46.608820 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:27:46.608887 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:27:46.609176 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:27:46.609177 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:27:46.609177 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:27:46.609315 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:27:46.609380 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 09:27:46.610724 (+ 1344us) server_negotiation.cc:299] Negotiation successful
0620 09:27:46.610813 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8653,"thread_start_us":78,"threads_started":1}
I20260620 09:27:46.611486 28003 heartbeater.cc:347] Connected to a master server at 127.16.43.190:34201
I20260620 09:27:46.611608 28003 heartbeater.cc:464] Registering TS with master...
I20260620 09:27:46.611805 28003 heartbeater.cc:511] Master 127.16.43.190:34201 requested a full tablet report, sending...
I20260620 09:27:46.612438 27791 ts_manager.cc:194] Registered new tserver with Master: d654c4732f754aca8098d35c4e1ae4bd (127.16.43.129:46487)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.43.130@KRBTEST.COM" created.
I20260620 09:27:46.613356 27791 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.43.129@KRBTEST.COM'} at 127.16.43.129:56719
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.43.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.130.keytab.
Entry for principal kudu/127.16.43.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.130.keytab.
I20260620 09:27:46.634516 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781947562949448-16558-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781947562949448-16558-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781947562949448-16558-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781947562949448-16558-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.43.148:53685/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.43.130:0
--local_ip_for_outbound_sockets=127.16.43.130
--webserver_interface=127.16.43.130
--webserver_port=0
--tserver_master_addrs=127.16.43.190:34201
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.130.keytab
--principal=kudu/127.16.43.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.43.148:42679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781947562949448-16558-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781947562949448-16558-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781947562949448-16558-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 09:27:46 dist-test-slave-706q krb5kdc[27315](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781947666, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260620 09:27:46.774312 28009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:27:46.774665 28009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:27:46.774914 28009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:27:46.776945 28009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:27:46.777046 28009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:27:46.777138 28009 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.43.130
I20260620 09:27:46.779472 28009 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:42679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781947562949448-16558-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781947562949448-16558-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.16.43.148:53685/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.130.keytab
--principal=kudu/127.16.43.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.43.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781947562949448-16558-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.43.130
--webserver_port=0
--tserver_master_addrs=127.16.43.190:34201
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28009
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.43.130
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781947562949448-16558-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:27:46.779997 28009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:27:46.780356 28009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:27:46.787256 28017 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:27:46.787561 28019 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:27:46.787693 28016 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:27:46.787817 28009 server_base.cc:1034] running on GCE node
I20260620 09:27:46.788141 28009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:27:46.788410 28009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:27:46.795724 28009 hybrid_clock.cc:648] HybridClock initialized: now 1781947666795690 us; error 42 us; skew 500 ppm
Jun 20 09:27:46 dist-test-slave-706q krb5kdc[27315](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947666, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:27:46.799171 28009 init.cc:377] Logged in from keytab as kudu/127.16.43.130@KRBTEST.COM (short username kudu)
I20260620 09:27:46.799741 28009 webserver.cc:460] Webserver started at http://127.16.43.130:39229/ using document root <none> and password file <none>
I20260620 09:27:46.800001 28009 fs_manager.cc:359] Metadata directory not provided
I20260620 09:27:46.800055 28009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:27:46.800181 28009 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 09:27:46 dist-test-slave-706q krb5kdc[27315](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947666, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.130@KRBTEST.COM for HTTP/127.16.43.148@KRBTEST.COM
I20260620 09:27:46.817320 28009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781947562949448-16558-0/minicluster-data/ts-1/data/instance:
uuid: "44a1de89e2a14ba29ff66e9f8714a250"
format_stamp: "Formatted at 2026-06-20 09:27:46 on dist-test-slave-706q"
server_key: "ed6759c92d8750bac3b3c8ac4822dde2"
server_key_iv: "e6d9446655e56eb266f0bfeb75d98312"
server_key_version: "kuduclusterkey@0"
I20260620 09:27:46.817902 28009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781947562949448-16558-0/minicluster-data/ts-1/wal/instance:
uuid: "44a1de89e2a14ba29ff66e9f8714a250"
format_stamp: "Formatted at 2026-06-20 09:27:46 on dist-test-slave-706q"
server_key: "ed6759c92d8750bac3b3c8ac4822dde2"
server_key_iv: "e6d9446655e56eb266f0bfeb75d98312"
server_key_version: "kuduclusterkey@0"
I20260620 09:27:46.820390 28009 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260620 09:27:46.836061 28026 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:27:46.836277 28009 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:27:46.836371 28009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781947562949448-16558-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781947562949448-16558-0/minicluster-data/ts-1/wal
uuid: "44a1de89e2a14ba29ff66e9f8714a250"
format_stamp: "Formatted at 2026-06-20 09:27:46 on dist-test-slave-706q"
server_key: "ed6759c92d8750bac3b3c8ac4822dde2"
server_key_iv: "e6d9446655e56eb266f0bfeb75d98312"
server_key_version: "kuduclusterkey@0"
I20260620 09:27:46.836467 28009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781947562949448-16558-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781947562949448-16558-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781947562949448-16558-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:27:46.873610 28009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:27:46.876305 28009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:27:46.876593 28009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:27:46.876888 28009 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:27:46.877305 28009 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:27:46.877416 28009 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:27:46.877496 28009 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:27:46.877555 28009 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:27:46.884299 28009 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.130:37907
I20260620 09:27:46.884780 28009 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781947562949448-16558-0/minicluster-data/ts-1/data/info.pb
I20260620 09:27:46.886613 28140 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.130:37907 every 8 connection(s)
I20260620 09:27:46.888460 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 28009
I20260620 09:27:46.888593 16558 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781947562949448-16558-0/minicluster-data/ts-1/wal/instance
I20260620 09:27:46.888836 16558 external_mini_cluster.cc:1428] Setting key c74d73e307ad7a90e999e2866208f7c8
Jun 20 09:27:46 dist-test-slave-706q krb5kdc[27315](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947666, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.130@KRBTEST.COM for kudu/127.16.43.190@KRBTEST.COM
I20260620 09:27:46.898495 28006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:27:46.886122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.130:53381 (local address 127.16.43.190:34201)
0620 09:27:46.886394 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0620 09:27:46.886397 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:27:46.887069 (+ 672us) server_negotiation.cc:407] Connection header received
0620 09:27:46.887836 (+ 767us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:27:46.887837 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:27:46.887855 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:27:46.887886 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:27:46.889749 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:27:46.890504 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:27:46.892034 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:27:46.892359 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:27:46.892410 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:27:46.895080 (+ 2670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:27:46.895082 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:27:46.895082 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:27:46.895090 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:27:46.896852 (+ 1762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:27:46.897369 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:27:46.897370 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:27:46.897371 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:27:46.897433 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:27:46.897731 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:27:46.897732 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:27:46.897732 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:27:46.897867 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:27:46.897911 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0620 09:27:46.898280 (+ 369us) server_negotiation.cc:299] Negotiation successful
0620 09:27:46.898385 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209}
WARNING: no policy specified for kudu/127.16.43.131@KRBTEST.COM; defaulting to no policy
I20260620 09:27:46.899235 28143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:27:46.886124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:34201 (local address 127.16.43.130:53381)
0620 09:27:46.886939 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0620 09:27:46.886979 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 09:27:46.887709 (+ 730us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:27:46.888095 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:27:46.888100 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:27:46.888435 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:27:46.889647 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:27:46.889654 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:27:46.890610 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:27:46.890611 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:27:46.891935 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:27:46.891938 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:27:46.892511 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:27:46.892511 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:27:46.892652 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:27:46.893262 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:27:46.893273 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:27:46.894973 (+ 1700us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:27:46.896953 (+ 1980us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:27:46.896960 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:27:46.896962 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:27:46.897259 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:27:46.897515 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:27:46.897516 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:27:46.897516 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:27:46.897653 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:27:46.897957 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:27:46.897963 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:27:46.898130 (+ 167us) client_negotiation.cc:769] Sending connection context
0620 09:27:46.898821 (+ 691us) client_negotiation.cc:240] Negotiation successful
0620 09:27:46.898998 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":774,"thread_start_us":80,"threads_started":1}
I20260620 09:27:46.900113 28141 heartbeater.cc:347] Connected to a master server at 127.16.43.190:34201
I20260620 09:27:46.900238 28141 heartbeater.cc:464] Registering TS with master...
I20260620 09:27:46.900455 28141 heartbeater.cc:511] Master 127.16.43.190:34201 requested a full tablet report, sending...
I20260620 09:27:46.901093 27791 ts_manager.cc:194] Registered new tserver with Master: 44a1de89e2a14ba29ff66e9f8714a250 (127.16.43.130:37907)
I20260620 09:27:46.901587 27791 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.43.130@KRBTEST.COM'} at 127.16.43.130:53381
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.43.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.43.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.131.keytab.
Entry for principal kudu/127.16.43.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.131.keytab.
I20260620 09:27:46.922254 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781947562949448-16558-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781947562949448-16558-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781947562949448-16558-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781947562949448-16558-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.43.148:53685/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.43.131:0
--local_ip_for_outbound_sockets=127.16.43.131
--webserver_interface=127.16.43.131
--webserver_port=0
--tserver_master_addrs=127.16.43.190:34201
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.131.keytab
--principal=kudu/127.16.43.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.43.148:42679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781947562949448-16558-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781947562949448-16558-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781947562949448-16558-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:27:47.042048 28146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:27:47.042394 28146 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:27:47.042456 28146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:27:47.044162 28146 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:27:47.044222 28146 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:27:47.044261 28146 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.43.131
I20260620 09:27:47.046368 28146 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:42679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781947562949448-16558-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781947562949448-16558-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.16.43.148:53685/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.131.keytab
--principal=kudu/127.16.43.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.43.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781947562949448-16558-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.43.131
--webserver_port=0
--tserver_master_addrs=127.16.43.190:34201
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28146
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.43.131
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781947562949448-16558-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:27:47.046643 28146 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:27:47.046924 28146 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:27:47.055863 28151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:27:47.055984 28152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:27:47.056152 28154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:27:47.057149 28146 server_base.cc:1034] running on GCE node
I20260620 09:27:47.057376 28146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:27:47.057662 28146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:27:47.058842 28146 hybrid_clock.cc:648] HybridClock initialized: now 1781947667058813 us; error 43 us; skew 500 ppm
Jun 20 09:27:47 dist-test-slave-706q krb5kdc[27315](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947667, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:27:47.062216 28146 init.cc:377] Logged in from keytab as kudu/127.16.43.131@KRBTEST.COM (short username kudu)
I20260620 09:27:47.062753 28146 webserver.cc:460] Webserver started at http://127.16.43.131:35931/ using document root <none> and password file <none>
I20260620 09:27:47.063313 28146 fs_manager.cc:359] Metadata directory not provided
I20260620 09:27:47.063400 28146 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:27:47.063572 28146 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 09:27:47 dist-test-slave-706q krb5kdc[27315](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947667, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.131@KRBTEST.COM for HTTP/127.16.43.148@KRBTEST.COM
I20260620 09:27:47.077142 28146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781947562949448-16558-0/minicluster-data/ts-2/data/instance:
uuid: "502bb99e68cb43d7b305fcdcce55cb18"
format_stamp: "Formatted at 2026-06-20 09:27:47 on dist-test-slave-706q"
server_key: "db0f5f808a5c9a29ed4f27049c523282"
server_key_iv: "e6a58c12fb7dcd381d40e6b87ae225b7"
server_key_version: "kuduclusterkey@0"
I20260620 09:27:47.077708 28146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781947562949448-16558-0/minicluster-data/ts-2/wal/instance:
uuid: "502bb99e68cb43d7b305fcdcce55cb18"
format_stamp: "Formatted at 2026-06-20 09:27:47 on dist-test-slave-706q"
server_key: "db0f5f808a5c9a29ed4f27049c523282"
server_key_iv: "e6a58c12fb7dcd381d40e6b87ae225b7"
server_key_version: "kuduclusterkey@0"
I20260620 09:27:47.079892 28146 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260620 09:27:47.104620 28161 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:27:47.105064 28146 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.001s sys 0.000s
I20260620 09:27:47.105166 28146 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781947562949448-16558-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781947562949448-16558-0/minicluster-data/ts-2/wal
uuid: "502bb99e68cb43d7b305fcdcce55cb18"
format_stamp: "Formatted at 2026-06-20 09:27:47 on dist-test-slave-706q"
server_key: "db0f5f808a5c9a29ed4f27049c523282"
server_key_iv: "e6a58c12fb7dcd381d40e6b87ae225b7"
server_key_version: "kuduclusterkey@0"
I20260620 09:27:47.105268 28146 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781947562949448-16558-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781947562949448-16558-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781947562949448-16558-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:27:47.127434 28146 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:27:47.130143 28146 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:27:47.130368 28146 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:27:47.130651 28146 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:27:47.131102 28146 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:27:47.131184 28146 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:27:47.131247 28146 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:27:47.131294 28146 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:27:47.138063 28146 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.131:45139
I20260620 09:27:47.138561 28146 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781947562949448-16558-0/minicluster-data/ts-2/data/info.pb
I20260620 09:27:47.142251 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 28146
I20260620 09:27:47.142364 16558 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781947562949448-16558-0/minicluster-data/ts-2/wal/instance
I20260620 09:27:47.142560 16558 external_mini_cluster.cc:1428] Setting key f12575aaa076b003c7650d2eb67818a8
I20260620 09:27:47.149124 28274 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.131:45139 every 8 connection(s)
Jun 20 09:27:47 dist-test-slave-706q krb5kdc[27315](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947667, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.131@KRBTEST.COM for kudu/127.16.43.190@KRBTEST.COM
I20260620 09:27:47.164270 28006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:27:47.152949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.131:40839 (local address 127.16.43.190:34201)
0620 09:27:47.153008 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0620 09:27:47.153010 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:27:47.153023 (+ 13us) server_negotiation.cc:407] Connection header received
0620 09:27:47.154134 (+ 1111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:27:47.154135 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:27:47.154151 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:27:47.154190 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:27:47.155357 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:27:47.156126 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:27:47.157639 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:27:47.157911 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:27:47.157963 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:27:47.160812 (+ 2849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:27:47.160814 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:27:47.160814 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:27:47.160823 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:27:47.162577 (+ 1754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:27:47.163075 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:27:47.163077 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:27:47.163077 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:27:47.163136 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:27:47.163435 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:27:47.163436 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:27:47.163437 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:27:47.163612 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:27:47.163659 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 09:27:47.164078 (+ 419us) server_negotiation.cc:299] Negotiation successful
0620 09:27:47.164173 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 09:27:47.165648 28277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:27:47.152598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:34201 (local address 127.16.43.131:40839)
0620 09:27:47.152862 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0620 09:27:47.152922 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0620 09:27:47.153608 (+ 686us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:27:47.154597 (+ 989us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:27:47.154602 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:27:47.154919 (+ 317us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:27:47.155228 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:27:47.155235 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:27:47.156257 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:27:47.156258 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:27:47.157501 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:27:47.157514 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:27:47.158074 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:27:47.158075 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:27:47.158228 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:27:47.158935 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:27:47.158946 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:27:47.160697 (+ 1751us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:27:47.162680 (+ 1983us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:27:47.162686 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:27:47.162689 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:27:47.162984 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:27:47.163206 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:27:47.163206 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:27:47.163207 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:27:47.163352 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:27:47.163707 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:27:47.163714 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:27:47.163886 (+ 172us) client_negotiation.cc:769] Sending connection context
0620 09:27:47.164032 (+ 146us) client_negotiation.cc:240] Negotiation successful
0620 09:27:47.164188 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":132,"threads_started":1}
I20260620 09:27:47.166460 28275 heartbeater.cc:347] Connected to a master server at 127.16.43.190:34201
I20260620 09:27:47.166585 28275 heartbeater.cc:464] Registering TS with master...
I20260620 09:27:47.166806 28275 heartbeater.cc:511] Master 127.16.43.190:34201 requested a full tablet report, sending...
I20260620 09:27:47.167227 27791 ts_manager.cc:194] Registered new tserver with Master: 502bb99e68cb43d7b305fcdcce55cb18 (127.16.43.131:45139)
I20260620 09:27:47.167747 27791 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.43.131@KRBTEST.COM'} at 127.16.43.131:40839
I20260620 09:27:47.176400 16558 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 09:27:47.186693 28006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:27:47.178206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58272 (local address 127.16.43.190:34201)
0620 09:27:47.178261 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 09:27:47.178263 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:27:47.178445 (+ 182us) server_negotiation.cc:407] Connection header received
0620 09:27:47.178498 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:27:47.178499 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:27:47.178513 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:27:47.178589 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:27:47.179354 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:27:47.180170 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:27:47.181258 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:27:47.181550 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:27:47.181632 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:27:47.183391 (+ 1759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:27:47.183393 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:27:47.183393 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:27:47.183401 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:27:47.185254 (+ 1853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:27:47.185717 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:27:47.185718 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:27:47.185718 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:27:47.185784 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:27:47.186005 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:27:47.186006 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:27:47.186006 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:27:47.186153 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:27:47.186234 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 09:27:47.186501 (+ 267us) server_negotiation.cc:299] Negotiation successful
0620 09:27:47.186592 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 09:27:47.190837 27791 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58272:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 09:27:47.193166 27791 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 09:27:47.201493 28288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:27:47.196203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.131:45139 (local address 127.0.0.1:42660)
0620 09:27:47.196737 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0620 09:27:47.196777 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 09:27:47.196850 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:27:47.197262 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:27:47.197265 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:27:47.197267 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:27:47.197349 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:27:47.197351 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:27:47.198589 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:27:47.198590 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:27:47.199932 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:27:47.199935 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:27:47.201180 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:27:47.201181 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:27:47.201263 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:27:47.201277 (+ 14us) client_negotiation.cc:769] Sending connection context
0620 09:27:47.201352 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 09:27:47.201404 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":68,"threads_started":1}
I20260620 09:27:47.201930 28291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:27:47.196464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42660 (local address 127.16.43.131:45139)
0620 09:27:47.196671 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 09:27:47.196676 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:27:47.196931 (+ 255us) server_negotiation.cc:407] Connection header received
0620 09:27:47.196973 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:27:47.196976 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:27:47.196991 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:27:47.197055 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:27:47.197422 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:27:47.198490 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:27:47.200050 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:27:47.200834 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:27:47.200907 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:27:47.200987 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 09:27:47.201805 (+ 818us) server_negotiation.cc:299] Negotiation successful
0620 09:27:47.201849 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":77,"threads_started":1}
I20260620 09:27:47.202929 28209 tablet_service.cc:1467] Processing CreateTablet for tablet d5c3580d317e4461b0cfbafe3bf06456 (DEFAULT_TABLE table=test-table [id=2bc92f405e8a4bcba74b94abd4dfd935]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:27:47.203300 28209 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5c3580d317e4461b0cfbafe3bf06456. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:27:47.209151 28294 tablet_bootstrap.cc:492] T d5c3580d317e4461b0cfbafe3bf06456 P 502bb99e68cb43d7b305fcdcce55cb18: Bootstrap starting.
I20260620 09:27:47.210347 28294 tablet_bootstrap.cc:654] T d5c3580d317e4461b0cfbafe3bf06456 P 502bb99e68cb43d7b305fcdcce55cb18: Neither blocks nor log segments found. Creating new log.
I20260620 09:27:47.212993 28290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:27:47.196222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.129:46487 (local address 127.0.0.1:55540)
0620 09:27:47.197154 (+ 932us) negotiation.cc:107] Waiting for socket to connect
0620 09:27:47.197164 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 09:27:47.197201 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:27:47.207268 (+ 10067us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:27:47.207270 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:27:47.207273 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:27:47.207330 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:27:47.207331 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:27:47.209617 (+ 2286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:27:47.209618 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:27:47.210845 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:27:47.210849 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:27:47.212750 (+ 1901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:27:47.212752 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:27:47.212818 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:27:47.212824 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 09:27:47.212859 (+ 35us) client_negotiation.cc:240] Negotiation successful
0620 09:27:47.212903 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":916,"thread_start_us":97,"threads_started":1}
I20260620 09:27:47.213099 28294 log.cc:824] T d5c3580d317e4461b0cfbafe3bf06456 P 502bb99e68cb43d7b305fcdcce55cb18: Log is configured to *not* fsync() on all Append() calls
I20260620 09:27:47.213464 28289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:27:47.196422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.130:37907 (local address 127.0.0.1:42938)
0620 09:27:47.201069 (+ 4647us) negotiation.cc:107] Waiting for socket to connect
0620 09:27:47.201080 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 09:27:47.201122 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:27:47.207628 (+ 6506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:27:47.207629 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:27:47.207631 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:27:47.207676 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:27:47.207677 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:27:47.210522 (+ 2845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:27:47.210523 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:27:47.211743 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:27:47.211746 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:27:47.213243 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:27:47.213244 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:27:47.213308 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:27:47.213312 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 09:27:47.213355 (+ 43us) client_negotiation.cc:240] Negotiation successful
0620 09:27:47.213393 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4628,"thread_start_us":65,"threads_started":1}
I20260620 09:27:47.214110 28292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:27:47.196725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55540 (local address 127.16.43.129:46487)
0620 09:27:47.207071 (+ 10346us) server_negotiation.cc:206] Beginning negotiation
0620 09:27:47.207077 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 09:27:47.207089 (+ 12us) server_negotiation.cc:407] Connection header received
0620 09:27:47.207126 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:27:47.207128 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:27:47.207147 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:27:47.207244 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:27:47.207441 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:27:47.208866 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:27:47.211569 (+ 2703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:27:47.212634 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:27:47.213884 (+ 1250us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:27:47.213961 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 09:27:47.213992 (+ 31us) server_negotiation.cc:299] Negotiation successful
0620 09:27:47.214034 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10303,"thread_start_us":64,"threads_started":1}
I20260620 09:27:47.214313 28293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:27:47.196903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42938 (local address 127.16.43.130:37907)
0620 09:27:47.207406 (+ 10503us) server_negotiation.cc:206] Beginning negotiation
0620 09:27:47.207411 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:27:47.207422 (+ 11us) server_negotiation.cc:407] Connection header received
0620 09:27:47.207460 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:27:47.207461 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:27:47.207476 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:27:47.207748 (+ 272us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:27:47.207828 (+ 80us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:27:47.208915 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:27:47.211889 (+ 2974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:27:47.212639 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:27:47.214067 (+ 1428us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:27:47.214154 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 09:27:47.214183 (+ 29us) server_negotiation.cc:299] Negotiation successful
0620 09:27:47.214229 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10460,"thread_start_us":93,"threads_started":1}
I20260620 09:27:47.214885 28294 tablet_bootstrap.cc:492] T d5c3580d317e4461b0cfbafe3bf06456 P 502bb99e68cb43d7b305fcdcce55cb18: No bootstrap required, opened a new log
I20260620 09:27:47.215219 28294 ts_tablet_manager.cc:1397] T d5c3580d317e4461b0cfbafe3bf06456 P 502bb99e68cb43d7b305fcdcce55cb18: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260620 09:27:47.215708 28056 tablet_service.cc:1467] Processing CreateTablet for tablet d5c3580d317e4461b0cfbafe3bf06456 (DEFAULT_TABLE table=test-table [id=2bc92f405e8a4bcba74b94abd4dfd935]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:27:47.216089 28056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5c3580d317e4461b0cfbafe3bf06456. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:27:47.217190 28294 raft_consensus.cc:348] T d5c3580d317e4461b0cfbafe3bf06456 P 502bb99e68cb43d7b305fcdcce55cb18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d654c4732f754aca8098d35c4e1ae4bd" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 46487 } } peers { permanent_uuid: "44a1de89e2a14ba29ff66e9f8714a250" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 37907 } } peers { permanent_uuid: "502bb99e68cb43d7b305fcdcce55cb18" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 45139 } }
I20260620 09:27:47.217360 28294 raft_consensus.cc:374] T d5c3580d317e4461b0cfbafe3bf06456 P 502bb99e68cb43d7b305fcdcce55cb18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:27:47.217389 28294 raft_consensus.cc:729] T d5c3580d317e4461b0cfbafe3bf06456 P 502bb99e68cb43d7b305fcdcce55cb18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 502bb99e68cb43d7b305fcdcce55cb18, State: Initialized, Role: FOLLOWER
I20260620 09:27:47.217525 28294 consensus_queue.cc:260] T d5c3580d317e4461b0cfbafe3bf06456 P 502bb99e68cb43d7b305fcdcce55cb18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d654c4732f754aca8098d35c4e1ae4bd" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 46487 } } peers { permanent_uuid: "44a1de89e2a14ba29ff66e9f8714a250" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 37907 } } peers { permanent_uuid: "502bb99e68cb43d7b305fcdcce55cb18" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 45139 } }
I20260620 09:27:47.217767 28294 ts_tablet_manager.cc:1428] T d5c3580d317e4461b0cfbafe3bf06456 P 502bb99e68cb43d7b305fcdcce55cb18: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 09:27:47.219581 28297 tablet_bootstrap.cc:492] T d5c3580d317e4461b0cfbafe3bf06456 P 44a1de89e2a14ba29ff66e9f8714a250: Bootstrap starting.
I20260620 09:27:47.219975 27937 tablet_service.cc:1467] Processing CreateTablet for tablet d5c3580d317e4461b0cfbafe3bf06456 (DEFAULT_TABLE table=test-table [id=2bc92f405e8a4bcba74b94abd4dfd935]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:27:47.220350 27937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5c3580d317e4461b0cfbafe3bf06456. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:27:47.220549 28297 tablet_bootstrap.cc:654] T d5c3580d317e4461b0cfbafe3bf06456 P 44a1de89e2a14ba29ff66e9f8714a250: Neither blocks nor log segments found. Creating new log.
I20260620 09:27:47.220810 28297 log.cc:824] T d5c3580d317e4461b0cfbafe3bf06456 P 44a1de89e2a14ba29ff66e9f8714a250: Log is configured to *not* fsync() on all Append() calls
I20260620 09:27:47.221482 28297 tablet_bootstrap.cc:492] T d5c3580d317e4461b0cfbafe3bf06456 P 44a1de89e2a14ba29ff66e9f8714a250: No bootstrap required, opened a new log
I20260620 09:27:47.221534 28297 ts_tablet_manager.cc:1397] T d5c3580d317e4461b0cfbafe3bf06456 P 44a1de89e2a14ba29ff66e9f8714a250: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 09:27:47.223320 28297 raft_consensus.cc:348] T d5c3580d317e4461b0cfbafe3bf06456 P 44a1de89e2a14ba29ff66e9f8714a250 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d654c4732f754aca8098d35c4e1ae4bd" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 46487 } } peers { permanent_uuid: "44a1de89e2a14ba29ff66e9f8714a250" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 37907 } } peers { permanent_uuid: "502bb99e68cb43d7b305fcdcce55cb18" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 45139 } }
I20260620 09:27:47.223483 28297 raft_consensus.cc:374] T d5c3580d317e4461b0cfbafe3bf06456 P 44a1de89e2a14ba29ff66e9f8714a250 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:27:47.223511 28297 raft_consensus.cc:729] T d5c3580d317e4461b0cfbafe3bf06456 P 44a1de89e2a14ba29ff66e9f8714a250 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44a1de89e2a14ba29ff66e9f8714a250, State: Initialized, Role: FOLLOWER
I20260620 09:27:47.223649 28297 consensus_queue.cc:260] T d5c3580d317e4461b0cfbafe3bf06456 P 44a1de89e2a14ba29ff66e9f8714a250 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d654c4732f754aca8098d35c4e1ae4bd" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 46487 } } peers { permanent_uuid: "44a1de89e2a14ba29ff66e9f8714a250" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 37907 } } peers { permanent_uuid: "502bb99e68cb43d7b305fcdcce55cb18" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 45139 } }
I20260620 09:27:47.223865 28297 ts_tablet_manager.cc:1428] T d5c3580d317e4461b0cfbafe3bf06456 P 44a1de89e2a14ba29ff66e9f8714a250: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 09:27:47.223915 28275 heartbeater.cc:503] Master 127.16.43.190:34201 was elected leader, sending a full tablet report...
I20260620 09:27:47.224007 28141 heartbeater.cc:503] Master 127.16.43.190:34201 was elected leader, sending a full tablet report...
I20260620 09:27:47.225750 28299 tablet_bootstrap.cc:492] T d5c3580d317e4461b0cfbafe3bf06456 P d654c4732f754aca8098d35c4e1ae4bd: Bootstrap starting.
I20260620 09:27:47.226562 28299 tablet_bootstrap.cc:654] T d5c3580d317e4461b0cfbafe3bf06456 P d654c4732f754aca8098d35c4e1ae4bd: Neither blocks nor log segments found. Creating new log.
I20260620 09:27:47.226956 28299 log.cc:824] T d5c3580d317e4461b0cfbafe3bf06456 P d654c4732f754aca8098d35c4e1ae4bd: Log is configured to *not* fsync() on all Append() calls
I20260620 09:27:47.227823 28299 tablet_bootstrap.cc:492] T d5c3580d317e4461b0cfbafe3bf06456 P d654c4732f754aca8098d35c4e1ae4bd: No bootstrap required, opened a new log
I20260620 09:27:47.227898 28299 ts_tablet_manager.cc:1397] T d5c3580d317e4461b0cfbafe3bf06456 P d654c4732f754aca8098d35c4e1ae4bd: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 09:27:47.230600 28299 raft_consensus.cc:348] T d5c3580d317e4461b0cfbafe3bf06456 P d654c4732f754aca8098d35c4e1ae4bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d654c4732f754aca8098d35c4e1ae4bd" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 46487 } } peers { permanent_uuid: "44a1de89e2a14ba29ff66e9f8714a250" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 37907 } } peers { permanent_uuid: "502bb99e68cb43d7b305fcdcce55cb18" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 45139 } }
I20260620 09:27:47.230912 28299 raft_consensus.cc:374] T d5c3580d317e4461b0cfbafe3bf06456 P d654c4732f754aca8098d35c4e1ae4bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:27:47.231195 28299 raft_consensus.cc:729] T d5c3580d317e4461b0cfbafe3bf06456 P d654c4732f754aca8098d35c4e1ae4bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d654c4732f754aca8098d35c4e1ae4bd, State: Initialized, Role: FOLLOWER
I20260620 09:27:47.231451 28299 consensus_queue.cc:260] T d5c3580d317e4461b0cfbafe3bf06456 P d654c4732f754aca8098d35c4e1ae4bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d654c4732f754aca8098d35c4e1ae4bd" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 46487 } } peers { permanent_uuid: "44a1de89e2a14ba29ff66e9f8714a250" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 37907 } } peers { permanent_uuid: "502bb99e68cb43d7b305fcdcce55cb18" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 45139 } }
I20260620 09:27:47.231815 28003 heartbeater.cc:503] Master 127.16.43.190:34201 was elected leader, sending a full tablet report...
I20260620 09:27:47.232051 28299 ts_tablet_manager.cc:1428] T d5c3580d317e4461b0cfbafe3bf06456 P d654c4732f754aca8098d35c4e1ae4bd: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
Jun 20 09:27:47 dist-test-slave-706q krb5kdc[27315](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947666, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.190@KRBTEST.COM for HTTP/127.16.43.148@KRBTEST.COM
W20260620 09:27:47.348392 28004 tablet.cc:2369] T d5c3580d317e4461b0cfbafe3bf06456 P d654c4732f754aca8098d35c4e1ae4bd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 09:27:47.386945 28142 tablet.cc:2369] T d5c3580d317e4461b0cfbafe3bf06456 P 44a1de89e2a14ba29ff66e9f8714a250: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 09:27:47.393785 28300 raft_consensus.cc:482] T d5c3580d317e4461b0cfbafe3bf06456 P 44a1de89e2a14ba29ff66e9f8714a250 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:27:47.393939 28300 raft_consensus.cc:504] T d5c3580d317e4461b0cfbafe3bf06456 P 44a1de89e2a14ba29ff66e9f8714a250 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d654c4732f754aca8098d35c4e1ae4bd" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 46487 } } peers { permanent_uuid: "44a1de89e2a14ba29ff66e9f8714a250" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 37907 } } peers { permanent_uuid: "502bb99e68cb43d7b305fcdcce55cb18" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 45139 } }
I20260620 09:27:47.394359 28300 leader_election.cc:288] T d5c3580d317e4461b0cfbafe3bf06456 P 44a1de89e2a14ba29ff66e9f8714a250 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d654c4732f754aca8098d35c4e1ae4bd (127.16.43.129:46487), 502bb99e68cb43d7b305fcdcce55cb18 (127.16.43.131:45139)
I20260620 09:27:47.398227 28291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:27:47.394772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.130:58867 (local address 127.16.43.131:45139)
0620 09:27:47.394814 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 09:27:47.394817 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:27:47.394827 (+ 10us) server_negotiation.cc:407] Connection header received
0620 09:27:47.394857 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:27:47.394858 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:27:47.394867 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:27:47.394934 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:27:47.395123 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:27:47.395889 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:27:47.397316 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:27:47.397865 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:27:47.397915 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:27:47.397925 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 09:27:47.398123 (+ 198us) server_negotiation.cc:299] Negotiation successful
0620 09:27:47.398158 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 09:27:47.398447 28303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:27:47.394573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.131:45139 (local address 127.16.43.130:58867)
0620 09:27:47.394723 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 09:27:47.394731 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 09:27:47.394774 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:27:47.394946 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:27:47.394947 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:27:47.394949 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:27:47.395033 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:27:47.395035 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:27:47.396007 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:27:47.396007 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:27:47.397209 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:27:47.397212 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:27:47.397984 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:27:47.397985 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:27:47.398051 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:27:47.398054 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 09:27:47.398307 (+ 253us) client_negotiation.cc:240] Negotiation successful
0620 09:27:47.398355 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":92,"threads_started":1}
I20260620 09:27:47.398859 28292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:27:47.395001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.130:59841 (local address 127.16.43.129:46487)
0620 09:27:47.395046 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 09:27:47.395048 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:27:47.395056 (+ 8us) server_negotiation.cc:407] Connection header received
0620 09:27:47.395081 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:27:47.395082 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:27:47.395090 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:27:47.395144 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:27:47.395365 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:27:47.396107 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:27:47.397514 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:27:47.398078 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:27:47.398152 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:27:47.398162 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 09:27:47.398750 (+ 588us) server_negotiation.cc:299] Negotiation successful
0620 09:27:47.398796 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 09:27:47.399073 28229 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5c3580d317e4461b0cfbafe3bf06456" candidate_uuid: "44a1de89e2a14ba29ff66e9f8714a250" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "502bb99e68cb43d7b305fcdcce55cb18" is_pre_election: true
I20260620 09:27:47.399281 28229 raft_consensus.cc:2454] T d5c3580d317e4461b0cfbafe3bf06456 P 502bb99e68cb43d7b305fcdcce55cb18 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44a1de89e2a14ba29ff66e9f8714a250 in term 0.
I20260620 09:27:47.399586 28143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:27:47.394545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.129:46487 (local address 127.16.43.130:59841)
0620 09:27:47.394591 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0620 09:27:47.394606 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 09:27:47.394655 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:27:47.395220 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:27:47.395222 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:27:47.395223 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:27:47.395286 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:27:47.395287 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:27:47.396292 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:27:47.396293 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:27:47.397385 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:27:47.397388 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:27:47.398194 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:27:47.398195 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:27:47.398288 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:27:47.398291 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 09:27:47.398348 (+ 57us) client_negotiation.cc:240] Negotiation successful
0620 09:27:47.398400 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260620 09:27:47.399641 28030 leader_election.cc:302] T d5c3580d317e4461b0cfbafe3bf06456 P 44a1de89e2a14ba29ff66e9f8714a250 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44a1de89e2a14ba29ff66e9f8714a250, 502bb99e68cb43d7b305fcdcce55cb18; no voters:
I20260620 09:27:47.399901 27957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5c3580d317e4461b0cfbafe3bf06456" candidate_uuid: "44a1de89e2a14ba29ff66e9f8714a250" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d654c4732f754aca8098d35c4e1ae4bd" is_pre_election: true
I20260620 09:27:47.400038 27957 raft_consensus.cc:2454] T d5c3580d317e4461b0cfbafe3bf06456 P d654c4732f754aca8098d35c4e1ae4bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44a1de89e2a14ba29ff66e9f8714a250 in term 0.
I20260620 09:27:47.400074 28300 raft_consensus.cc:2781] T d5c3580d317e4461b0cfbafe3bf06456 P 44a1de89e2a14ba29ff66e9f8714a250 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 09:27:47.400141 28300 raft_consensus.cc:482] T d5c3580d317e4461b0cfbafe3bf06456 P 44a1de89e2a14ba29ff66e9f8714a250 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 09:27:47.400170 28300 raft_consensus.cc:3037] T d5c3580d317e4461b0cfbafe3bf06456 P 44a1de89e2a14ba29ff66e9f8714a250 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:27:47.401105 28300 raft_consensus.cc:504] T d5c3580d317e4461b0cfbafe3bf06456 P 44a1de89e2a14ba29ff66e9f8714a250 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d654c4732f754aca8098d35c4e1ae4bd" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 46487 } } peers { permanent_uuid: "44a1de89e2a14ba29ff66e9f8714a250" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 37907 } } peers { permanent_uuid: "502bb99e68cb43d7b305fcdcce55cb18" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 45139 } }
I20260620 09:27:47.401265 28300 leader_election.cc:288] T d5c3580d317e4461b0cfbafe3bf06456 P 44a1de89e2a14ba29ff66e9f8714a250 [CANDIDATE]: Term 1 election: Requested vote from peers d654c4732f754aca8098d35c4e1ae4bd (127.16.43.129:46487), 502bb99e68cb43d7b305fcdcce55cb18 (127.16.43.131:45139)
I20260620 09:27:47.401453 27957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5c3580d317e4461b0cfbafe3bf06456" candidate_uuid: "44a1de89e2a14ba29ff66e9f8714a250" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d654c4732f754aca8098d35c4e1ae4bd"
I20260620 09:27:47.401531 27957 raft_consensus.cc:3037] T d5c3580d317e4461b0cfbafe3bf06456 P d654c4732f754aca8098d35c4e1ae4bd [term 0 FOLLOWER]: Advancing to term 1
W20260620 09:27:47.402135 28276 tablet.cc:2369] T d5c3580d317e4461b0cfbafe3bf06456 P 502bb99e68cb43d7b305fcdcce55cb18: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 09:27:47.402433 27957 raft_consensus.cc:2454] T d5c3580d317e4461b0cfbafe3bf06456 P d654c4732f754aca8098d35c4e1ae4bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44a1de89e2a14ba29ff66e9f8714a250 in term 1.
I20260620 09:27:47.402643 28029 leader_election.cc:302] T d5c3580d317e4461b0cfbafe3bf06456 P 44a1de89e2a14ba29ff66e9f8714a250 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44a1de89e2a14ba29ff66e9f8714a250, d654c4732f754aca8098d35c4e1ae4bd; no voters:
I20260620 09:27:47.402794 28229 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5c3580d317e4461b0cfbafe3bf06456" candidate_uuid: "44a1de89e2a14ba29ff66e9f8714a250" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "502bb99e68cb43d7b305fcdcce55cb18"
I20260620 09:27:47.402884 28229 raft_consensus.cc:3037] T d5c3580d317e4461b0cfbafe3bf06456 P 502bb99e68cb43d7b305fcdcce55cb18 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:27:47.403784 28229 raft_consensus.cc:2454] T d5c3580d317e4461b0cfbafe3bf06456 P 502bb99e68cb43d7b305fcdcce55cb18 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44a1de89e2a14ba29ff66e9f8714a250 in term 1.
I20260620 09:27:47.403954 28300 raft_consensus.cc:2781] T d5c3580d317e4461b0cfbafe3bf06456 P 44a1de89e2a14ba29ff66e9f8714a250 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:27:47.404119 28300 raft_consensus.cc:686] T d5c3580d317e4461b0cfbafe3bf06456 P 44a1de89e2a14ba29ff66e9f8714a250 [term 1 LEADER]: Becoming Leader. State: Replica: 44a1de89e2a14ba29ff66e9f8714a250, State: Running, Role: LEADER
I20260620 09:27:47.404220 28300 consensus_queue.cc:237] T d5c3580d317e4461b0cfbafe3bf06456 P 44a1de89e2a14ba29ff66e9f8714a250 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d654c4732f754aca8098d35c4e1ae4bd" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 46487 } } peers { permanent_uuid: "44a1de89e2a14ba29ff66e9f8714a250" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 37907 } } peers { permanent_uuid: "502bb99e68cb43d7b305fcdcce55cb18" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 45139 } }
I20260620 09:27:47.405133 27791 catalog_manager.cc:5495] T d5c3580d317e4461b0cfbafe3bf06456 P 44a1de89e2a14ba29ff66e9f8714a250 reported cstate change: term changed from 0 to 1, leader changed from <none> to 44a1de89e2a14ba29ff66e9f8714a250 (127.16.43.130). New cstate: current_term: 1 leader_uuid: "44a1de89e2a14ba29ff66e9f8714a250" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d654c4732f754aca8098d35c4e1ae4bd" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 46487 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44a1de89e2a14ba29ff66e9f8714a250" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 37907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "502bb99e68cb43d7b305fcdcce55cb18" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 45139 } health_report { overall_health: UNKNOWN } } }
I20260620 09:27:47.436872 28293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:27:47.433505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42950 (local address 127.16.43.130:37907)
0620 09:27:47.433582 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0620 09:27:47.433585 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:27:47.433595 (+ 10us) server_negotiation.cc:407] Connection header received
0620 09:27:47.433642 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:27:47.433643 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:27:47.433665 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:27:47.433755 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:27:47.433924 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:27:47.434678 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:27:47.435790 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:27:47.436079 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:27:47.436163 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:27:47.436352 (+ 189us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:27:47.436495 (+ 143us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:27:47.436589 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 09:27:47.436768 (+ 179us) server_negotiation.cc:299] Negotiation successful
0620 09:27:47.436788 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 09:27:47.439204 27957 raft_consensus.cc:1261] T d5c3580d317e4461b0cfbafe3bf06456 P d654c4732f754aca8098d35c4e1ae4bd [term 1 FOLLOWER]: Refusing update from remote peer 44a1de89e2a14ba29ff66e9f8714a250: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:27:47.439623 28229 raft_consensus.cc:1261] T d5c3580d317e4461b0cfbafe3bf06456 P 502bb99e68cb43d7b305fcdcce55cb18 [term 1 FOLLOWER]: Refusing update from remote peer 44a1de89e2a14ba29ff66e9f8714a250: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:27:47.439949 28304 consensus_queue.cc:1035] T d5c3580d317e4461b0cfbafe3bf06456 P 44a1de89e2a14ba29ff66e9f8714a250 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d654c4732f754aca8098d35c4e1ae4bd" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 46487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:27:47.440140 28300 consensus_queue.cc:1035] T d5c3580d317e4461b0cfbafe3bf06456 P 44a1de89e2a14ba29ff66e9f8714a250 [LEADER]: Connected to new peer: Peer: permanent_uuid: "502bb99e68cb43d7b305fcdcce55cb18" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:27:47.445999 28307 mvcc.cc:205] Tried to move back new op lower bound from 7298857645827862528 to 7298857645688291328. Current Snapshot: MvccSnapshot[applied={T|T < 7298857645827862528}]
I20260620 09:27:47.446206 28308 mvcc.cc:205] Tried to move back new op lower bound from 7298857645827862528 to 7298857645688291328. Current Snapshot: MvccSnapshot[applied={T|T < 7298857645827862528}]
I20260620 09:27:47.446427 28310 mvcc.cc:205] Tried to move back new op lower bound from 7298857645827862528 to 7298857645688291328. Current Snapshot: MvccSnapshot[applied={T|T < 7298857645827862528}]
I20260620 09:27:47.450665 27789 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58272:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:27:47.450773 27789 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58272:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:27:47.451740 27789 catalog_manager.cc:5782] T 00000000000000000000000000000000 P be5fa4a54bc9438789b0b97171758dc6: Sending DeleteTablet for 3 replicas of tablet d5c3580d317e4461b0cfbafe3bf06456
I20260620 09:27:47.452232 28209 tablet_service.cc:1514] Processing DeleteTablet for tablet d5c3580d317e4461b0cfbafe3bf06456 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:27:47 UTC) from {username='kudu', principal='kudu/127.16.43.190@KRBTEST.COM'} at 127.0.0.1:42660
I20260620 09:27:47.452483 27937 tablet_service.cc:1514] Processing DeleteTablet for tablet d5c3580d317e4461b0cfbafe3bf06456 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:27:47 UTC) from {username='kudu', principal='kudu/127.16.43.190@KRBTEST.COM'} at 127.0.0.1:55540
I20260620 09:27:47.452677 28056 tablet_service.cc:1514] Processing DeleteTablet for tablet d5c3580d317e4461b0cfbafe3bf06456 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:27:47 UTC) from {username='kudu', principal='kudu/127.16.43.190@KRBTEST.COM'} at 127.0.0.1:42938
I20260620 09:27:47.453450 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 27872
I20260620 09:27:47.455938 28319 tablet_replica.cc:330] T d5c3580d317e4461b0cfbafe3bf06456 P 44a1de89e2a14ba29ff66e9f8714a250: stopping tablet replica
I20260620 09:27:47.459916 28319 raft_consensus.cc:2229] T d5c3580d317e4461b0cfbafe3bf06456 P 44a1de89e2a14ba29ff66e9f8714a250 [term 1 LEADER]: Raft consensus shutting down.
I20260620 09:27:47.460217 28319 raft_consensus.cc:2258] T d5c3580d317e4461b0cfbafe3bf06456 P 44a1de89e2a14ba29ff66e9f8714a250 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:27:47.460841 28319 ts_tablet_manager.cc:1905] T d5c3580d317e4461b0cfbafe3bf06456 P 44a1de89e2a14ba29ff66e9f8714a250: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:27:47.462716 28319 ts_tablet_manager.cc:1918] T d5c3580d317e4461b0cfbafe3bf06456 P 44a1de89e2a14ba29ff66e9f8714a250: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:27:47.462858 28319 log.cc:1196] T d5c3580d317e4461b0cfbafe3bf06456 P 44a1de89e2a14ba29ff66e9f8714a250: Deleting WAL directory at /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781947562949448-16558-0/minicluster-data/ts-1/wal/wals/d5c3580d317e4461b0cfbafe3bf06456
I20260620 09:27:47.463207 28319 ts_tablet_manager.cc:1939] T d5c3580d317e4461b0cfbafe3bf06456 P 44a1de89e2a14ba29ff66e9f8714a250: Deleting consensus metadata
I20260620 09:27:47.463763 27777 catalog_manager.cc:4841] TS 44a1de89e2a14ba29ff66e9f8714a250 (127.16.43.130:37907): tablet d5c3580d317e4461b0cfbafe3bf06456 (table test-table [id=2bc92f405e8a4bcba74b94abd4dfd935]) successfully deleted
W20260620 09:27:47.464139 27777 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.43.129:46487 (error 108)
I20260620 09:27:47.464769 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 28009
I20260620 09:27:47.467334 28317 tablet_replica.cc:330] T d5c3580d317e4461b0cfbafe3bf06456 P 502bb99e68cb43d7b305fcdcce55cb18: stopping tablet replica
I20260620 09:27:47.467486 28317 raft_consensus.cc:2229] T d5c3580d317e4461b0cfbafe3bf06456 P 502bb99e68cb43d7b305fcdcce55cb18 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:27:47.467643 28317 raft_consensus.cc:2258] T d5c3580d317e4461b0cfbafe3bf06456 P 502bb99e68cb43d7b305fcdcce55cb18 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:27:47.467970 28289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:27:47.467846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.129:46487 (local address 127.0.0.1:55550)
0620 09:27:47.467897 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0620 09:27:47.467933 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.129:46487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260620 09:27:47.468084 27777 catalog_manager.cc:4584] TS d654c4732f754aca8098d35c4e1ae4bd (127.16.43.129:46487): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d5c3580d317e4461b0cfbafe3bf06456: Network error: Client connection negotiation failed: client connection to 127.16.43.129:46487: connect: Connection refused (error 111)
I20260620 09:27:47.468448 28317 ts_tablet_manager.cc:1905] T d5c3580d317e4461b0cfbafe3bf06456 P 502bb99e68cb43d7b305fcdcce55cb18: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:27:47.470618 28317 ts_tablet_manager.cc:1918] T d5c3580d317e4461b0cfbafe3bf06456 P 502bb99e68cb43d7b305fcdcce55cb18: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:27:47.470754 28317 log.cc:1196] T d5c3580d317e4461b0cfbafe3bf06456 P 502bb99e68cb43d7b305fcdcce55cb18: Deleting WAL directory at /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781947562949448-16558-0/minicluster-data/ts-2/wal/wals/d5c3580d317e4461b0cfbafe3bf06456
I20260620 09:27:47.471158 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 28146
I20260620 09:27:47.475711 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 27759
I20260620 09:27:47.475837 28289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:27:47.475694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.131:45139 (local address 127.0.0.1:42666)
0620 09:27:47.475759 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0620 09:27:47.475802 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.131:45139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260620 09:27:47.481482 16558 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 09:27:47.826721 16558 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 09:27:47.826877 16558 mini_ranger.cc:66] Stopping Ranger...
I20260620 09:27:48.172730 16558 mini_ranger.cc:68] Stopped Ranger
2026-06-20 09:27:48.173 UTC [27345] LOG: received smart shutdown request
2026-06-20 09:27:48.176 UTC [27345] LOG: background worker "logical replication launcher" (PID 27354) exited with exit code 1
2026-06-20 09:27:48.177 UTC [27349] LOG: shutting down
2026-06-20 09:27:48.356 UTC [27345] LOG: database system is shut down
2026-06-20T09:27:48Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (47970 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:27:48 dist-test-slave-706q krb5kdc[28329](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:27:48 dist-test-slave-706q krb5kdc[28329](info): set up 2 sockets
Jun 20 09:27:48 dist-test-slave-706q krb5kdc[28329](info): commencing operation
krb5kdc: starting...
W20260620 09:27:49.584632 16558 mini_kdc.cc:121] Time spent starting KDC: real 1.082s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/krb5kdc/test-user.keytab.
Jun 20 09:27:49 dist-test-slave-706q krb5kdc[28329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947669, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:27:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:27:49Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.16.43.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.16.43.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.16.43.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/krb5kdc/rangeradmin_127.16.43.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.16.43.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/krb5kdc/rangeradmin_127.16.43.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.16.43.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.16.43.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.16.43.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/krb5kdc/rangerlookup_127.16.43.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.16.43.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/krb5kdc/rangerlookup_127.16.43.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.16.43.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.16.43.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.16.43.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/krb5kdc/HTTP_127.16.43.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.16.43.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/krb5kdc/HTTP_127.16.43.148@KRBTEST.COM.keytab.
I20260620 09:27:49.758417 16558 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskMkdNwY/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/postgres -l logfile start
2026-06-20 09:27:52.481 UTC [28359] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 09:27:52.481 UTC [28359] LOG: listening on IPv4 address "127.16.43.148", port 33673
2026-06-20 09:27:52.485 UTC [28359] LOG: listening on Unix socket "/tmp/.s.PGSQL.33673"
2026-06-20 09:27:52.518 UTC [28362] LOG: database system was shut down at 2026-06-20 09:27:50 UTC
2026-06-20 09:27:52.522 UTC [28359] LOG: database system is ready to accept connections
I20260620 09:27:53.476404 16558 mini_postgres.cc:96] Postgres bound to 33673
2026-06-20 09:27:53.482 UTC [28370] FATAL: database "slave" does not exist
127.16.43.148:33673 - accepting connections
I20260620 09:27:53.483049 16558 mini_ranger.cc:161] Starting Ranger...
I20260620 09:27:53.497658 16558 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 09:27:54.024291 16558 mini_ranger.cc:87] Created ranger Postgres database
I20260620 09:27:54.024377 16558 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/ranger-admin
2026-06-20 09:27:54,299 [I] DB FLAVOR :POSTGRES
2026-06-20 09:27:54,300 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 09:27:54,300 [I] Checking connection..
2026-06-20 09:27:54,300 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMkdNwY/build/release/bin/postgresql.jar:/tmp/dist-test-taskMkdNwY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.43.148:33673/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 09:27:54,553 [I] Checking connection passed.
2026-06-20 09:27:54,553 [I] --------- Verifying version history table ---------
2026-06-20 09:27:54,554 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMkdNwY/build/release/bin/postgresql.jar:/tmp/dist-test-taskMkdNwY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.43.148:33673/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 09:27:54,810 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 09:27:54,810 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMkdNwY/build/release/bin/postgresql.jar:/tmp/dist-test-taskMkdNwY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.43.148:33673/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 09:27:55,060 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 09:27:55,060 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 09:27:55,060 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMkdNwY/build/release/bin/postgresql.jar:/tmp/dist-test-taskMkdNwY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.43.148:33673/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskMkdNwY/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 09:27:55.305 UTC [28434] WARNING: there is no transaction in progress
2026-06-20 09:27:55,311 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 09:27:55,312 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMkdNwY/build/release/bin/postgresql.jar:/tmp/dist-test-taskMkdNwY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.43.148:33673/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 09:27:55,562 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 09:27:55,562 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 09:27:55,563 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMkdNwY/build/release/bin/postgresql.jar:/tmp/dist-test-taskMkdNwY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.43.148:33673/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 09:27:55,796 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMkdNwY/build/release/bin/postgresql.jar:/tmp/dist-test-taskMkdNwY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.43.148:33673/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 09:27:56,041 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMkdNwY/build/release/bin/postgresql.jar:/tmp/dist-test-taskMkdNwY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.43.148:33673/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-706q.c.gcp-upstream.internal','N') ;"
2026-06-20 09:27:56,273 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMkdNwY/build/release/bin/postgresql.jar:/tmp/dist-test-taskMkdNwY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.43.148:33673/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 09:27:56,512 [I] Table x_portal_user does not exist in database ranger
2026-06-20 09:27:56,512 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMkdNwY/build/release/bin/postgresql.jar:/tmp/dist-test-taskMkdNwY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.43.148:33673/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 09:27:56,747 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 09:27:56,747 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMkdNwY/build/release/bin/postgresql.jar:/tmp/dist-test-taskMkdNwY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.43.148:33673/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 09:27:56,990 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMkdNwY/build/release/bin/postgresql.jar:/tmp/dist-test-taskMkdNwY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.43.148:33673/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 09:27:57,238 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 09:27:57,239 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMkdNwY/build/release/bin/postgresql.jar:/tmp/dist-test-taskMkdNwY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.43.148:33673/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskMkdNwY/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 09:27:58.237 UTC [28542] WARNING: there is no transaction in progress
2026-06-20 09:27:58.255 UTC [28542] WARNING: there is no transaction in progress
2026-06-20 09:27:58.274 UTC [28542] WARNING: there is no transaction in progress
2026-06-20 09:27:58.291 UTC [28542] WARNING: there is no transaction in progress
2026-06-20 09:27:58.307 UTC [28542] WARNING: there is no transaction in progress
2026-06-20 09:27:58.326 UTC [28542] WARNING: there is no transaction in progress
2026-06-20 09:27:58.384 UTC [28542] WARNING: there is no transaction in progress
2026-06-20 09:27:58.391 UTC [28542] WARNING: there is no transaction in progress
2026-06-20 09:27:58.411 UTC [28542] WARNING: there is no transaction in progress
2026-06-20 09:27:58.423 UTC [28542] WARNING: there is no transaction in progress
2026-06-20 09:27:58.434 UTC [28542] WARNING: there is no transaction in progress
2026-06-20 09:27:58.448 UTC [28542] WARNING: there is no transaction in progress
2026-06-20 09:27:58.459 UTC [28542] WARNING: there is no transaction in progress
2026-06-20 09:27:58.466 UTC [28542] WARNING: there is no transaction in progress
2026-06-20 09:27:58.473 UTC [28542] WARNING: there is no transaction in progress
2026-06-20 09:27:58.479 UTC [28542] WARNING: there is no transaction in progress
2026-06-20 09:27:59,167 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 09:27:59,167 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMkdNwY/build/release/bin/postgresql.jar:/tmp/dist-test-taskMkdNwY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.43.148:33673/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 09:27:59,399 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 09:27:59,399 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMkdNwY/build/release/bin/postgresql.jar:/tmp/dist-test-taskMkdNwY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.43.148:33673/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 09:27:59,651 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 09:27:59,652 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMkdNwY/build/release/bin/postgresql.jar:/tmp/dist-test-taskMkdNwY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.43.148:33673/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 09:27:59,914 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 09:27:59,914 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMkdNwY/build/release/bin/postgresql.jar:/tmp/dist-test-taskMkdNwY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.43.148:33673/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 09:28:00,177 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMkdNwY/build/release/bin/postgresql.jar:/tmp/dist-test-taskMkdNwY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.43.148:33673/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 09:28:00,433 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMkdNwY/build/release/bin/postgresql.jar:/tmp/dist-test-taskMkdNwY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.43.148:33673/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-706q.c.gcp-upstream.internal';"
2026-06-20 09:28:00,692 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 09:28:00,692 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMkdNwY/build/release/bin/postgresql.jar:/tmp/dist-test-taskMkdNwY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.43.148:33673/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 09:28:00,932 [I] DB_PATCHES have already been applied
I20260620 09:28:00.937825 16558 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/ranger-admin:/tmp/dist-test-taskMkdNwY/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskMkdNwY/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskMkdNwY/build/release/bin/postgresql.jar:/tmp/dist-test-taskMkdNwY/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 09:28:00.937894 16558 mini_ranger.cc:193] Using host: 127.16.43.148
I20260620 09:28:00.941049 16558 mini_ranger.cc:227] Ranger admin URL: http://127.16.43.148:51971
Jun 20, 2026 9:28:01 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 9:28:01 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 9:28:01 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 9:28:01 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 9:28:01 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 9:28:01 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 9:28:01 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 09:28:02.055143 16558 mini_ranger.cc:160] Time spent starting Ranger: real 8.572s user 0.000s sys 0.009s
Jun 20, 2026 9:28:21 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 9:28:21 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 20, 2026 9:28:21 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 9:28:21 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 9:28:21 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 9:28:21 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 9:28:21 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 9:28:21 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 9:28:21 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 9:28:21 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 9:28:21 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 9:28:21 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 9:28:21 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 9:28:21 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 9:28:21 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 9:28:21 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 9:28:21 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 9:28:22 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 09:28:23.332314 16558 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.16.43.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.16.43.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.16.43.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/krb5kdc/rangerkms_127.16.43.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.16.43.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/krb5kdc/rangerkms_127.16.43.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.16.43.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.16.43.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.16.43.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/krb5kdc/HTTP_127.16.43.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.16.43.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/krb5kdc/HTTP_127.16.43.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 09:28:23 dist-test-slave-706q krb5kdc[28329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947703, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 09:28:23.417770 16558 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 09:28:23.427124 16558 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 09:28:24.295310 16558 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 09:28:24.295399 16558 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/ranger-kms
I20260620 09:28:24.295418 16558 mini_ranger_kms.cc:227] Using postgres at 127.16.43.148:33673
2026-06-20 09:28:24,477 [I] DB FLAVOR :POSTGRES
2026-06-20 09:28:24,477 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 09:28:24,477 [I] Checking connection
2026-06-20 09:28:24,478 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMkdNwY/build/release/bin/postgresql.jar:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.43.148:33673/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 09:28:24,746 [I] connection success
2026-06-20 09:28:24,746 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 09:28:24,747 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 09:28:24,747 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMkdNwY/build/release/bin/postgresql.jar:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.43.148:33673/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 09:28:25,018 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 09:28:25,019 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 09:28:25,019 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 09:28:25,019 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMkdNwY/build/release/bin/postgresql.jar:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.43.148:33673/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 09:28:25,341 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 09:28:25.841238 16558 mini_ranger_kms.cc:310] Created kms service
I20260620 09:28:26.057842 16558 mini_ranger_kms.cc:326] Created kudu user
I20260620 09:28:26.108500 16558 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 09:28:26 dist-test-slave-706q krb5kdc[28329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947703, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.16.43.148@KRBTEST.COM
I20260620 09:28:26.519778 16558 mini_ranger_kms.cc:384] Added ranger policy
I20260620 09:28:26.519992 16558 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/ranger-kms:/tmp/dist-test-taskMkdNwY/build/release/bin/postgresql.jar:/tmp/dist-test-taskMkdNwY/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskMkdNwY/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 09:28:26.520016 16558 mini_ranger_kms.cc:240] Using host: 127.16.43.148
I20260620 09:28:26.523357 16558 mini_ranger_kms.cc:276] Ranger KMS PID: 28722
I20260620 09:28:26.523443 16558 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.16.43.148:34337
Jun 20, 2026 9:28:27 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 9:28:27 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 9:28:28 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 9:28:28 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 9:28:28 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 9:28:28 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260620 09:28:28.860606 16558 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.443s user 0.012s sys 0.151s
I20260620 09:28:28.860710 16558 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 09:28:28.860756 16558 mini_ranger_kms.cc:401] 127.16.43.148:34337/kms/v1/keys
Jun 20 09:28:29 dist-test-slave-706q krb5kdc[28329](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781947709, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.16.43.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 09:28:29 dist-test-slave-706q krb5kdc[28329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947709, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.16.43.148@KRBTEST.COM for HTTP/127.16.43.148@KRBTEST.COM
Jun 20 09:28:30 dist-test-slave-706q krb5kdc[28329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947709, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.16.43.148@KRBTEST.COM for HTTP/127.16.43.148@KRBTEST.COM
Jun 20 09:28:32 dist-test-slave-706q krb5kdc[28329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947712, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.16.43.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.43.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.43.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.190.keytab.
Entry for principal kudu/127.16.43.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.190.keytab.
I20260620 09:28:32.502195 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.43.190:36483
--webserver_interface=127.16.43.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.190.keytab
--principal=kudu/127.16.43.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.43.148:34845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.43.190:36483
--ranger_config_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.43.148:34337/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:28:32.611421 28773 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:28:32.611734 28773 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:28:32.611771 28773 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:28:32.612982 28773 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:28:32.613061 28773 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:28:32.613086 28773 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:28:32.613121 28773 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:28:32.613168 28773 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:28:32.614805 28773 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:34845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/master-0/wal
--ranger_kms_url=127.16.43.148:34337/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.16.43.190:36483
--ranger_config_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.190.keytab
--principal=kudu/127.16.43.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.43.190:36483
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.43.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28773
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:28:32.615108 28773 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:28:32.615342 28773 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:28:32.621244 28779 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:28:32.621263 28781 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:28:32.621330 28773 server_base.cc:1034] running on GCE node
W20260620 09:28:32.621399 28778 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:28:32.621709 28773 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:28:32.622040 28773 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:28:32.623200 28773 hybrid_clock.cc:648] HybridClock initialized: now 1781947712623178 us; error 39 us; skew 500 ppm
Jun 20 09:28:32 dist-test-slave-706q krb5kdc[28329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947712, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:28:32.625780 28773 init.cc:377] Logged in from keytab as kudu/127.16.43.190@KRBTEST.COM (short username kudu)
I20260620 09:28:32.626298 28773 webserver.cc:460] Webserver started at http://127.16.43.190:33705/ using document root <none> and password file <none>
I20260620 09:28:32.626570 28773 fs_manager.cc:359] Metadata directory not provided
I20260620 09:28:32.626623 28773 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:28:32.626767 28773 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 09:28:32 dist-test-slave-706q krb5kdc[28329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947712, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.190@KRBTEST.COM for HTTP/127.16.43.148@KRBTEST.COM
I20260620 09:28:32.681931 28773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/master-0/data/instance:
uuid: "cf7a3184e51948d7865e7f0756fc510a"
format_stamp: "Formatted at 2026-06-20 09:28:32 on dist-test-slave-706q"
server_key: "7c4542e727ca8934624a1050d9f5b50e"
server_key_iv: "d55969e6284df3de3136e7a71e9741d9"
server_key_version: "kuduclusterkey@0"
I20260620 09:28:32.682396 28773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/master-0/wal/instance:
uuid: "cf7a3184e51948d7865e7f0756fc510a"
format_stamp: "Formatted at 2026-06-20 09:28:32 on dist-test-slave-706q"
server_key: "7c4542e727ca8934624a1050d9f5b50e"
server_key_iv: "d55969e6284df3de3136e7a71e9741d9"
server_key_version: "kuduclusterkey@0"
I20260620 09:28:32.684350 28773 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.002s
I20260620 09:28:32.700932 28788 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:28:32.701220 28773 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:28:32.701331 28773 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/master-0/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/master-0/wal
uuid: "cf7a3184e51948d7865e7f0756fc510a"
format_stamp: "Formatted at 2026-06-20 09:28:32 on dist-test-slave-706q"
server_key: "7c4542e727ca8934624a1050d9f5b50e"
server_key_iv: "d55969e6284df3de3136e7a71e9741d9"
server_key_version: "kuduclusterkey@0"
I20260620 09:28:32.701433 28773 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:28:32.728694 28773 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:28:32.731192 28773 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:28:32.731351 28773 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:28:32.735770 28773 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.190:36483
I20260620 09:28:32.735808 28840 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.190:36483 every 8 connection(s)
I20260620 09:28:32.736189 28773 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
I20260620 09:28:32.736984 28841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:28:32.739461 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 28773
I20260620 09:28:32.739614 16558 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/master-0/wal/instance
I20260620 09:28:32.739825 16558 external_mini_cluster.cc:1428] Setting key 566f68cd0de0a31e48603a7af3df9f24
I20260620 09:28:32.740036 28841 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf7a3184e51948d7865e7f0756fc510a: Bootstrap starting.
I20260620 09:28:32.740759 28841 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cf7a3184e51948d7865e7f0756fc510a: Neither blocks nor log segments found. Creating new log.
I20260620 09:28:32.741079 28841 log.cc:824] T 00000000000000000000000000000000 P cf7a3184e51948d7865e7f0756fc510a: Log is configured to *not* fsync() on all Append() calls
I20260620 09:28:32.741837 28841 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf7a3184e51948d7865e7f0756fc510a: No bootstrap required, opened a new log
I20260620 09:28:32.743824 28841 raft_consensus.cc:348] T 00000000000000000000000000000000 P cf7a3184e51948d7865e7f0756fc510a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf7a3184e51948d7865e7f0756fc510a" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 36483 } }
I20260620 09:28:32.743996 28841 raft_consensus.cc:374] T 00000000000000000000000000000000 P cf7a3184e51948d7865e7f0756fc510a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:28:32.744026 28841 raft_consensus.cc:729] T 00000000000000000000000000000000 P cf7a3184e51948d7865e7f0756fc510a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf7a3184e51948d7865e7f0756fc510a, State: Initialized, Role: FOLLOWER
I20260620 09:28:32.744203 28841 consensus_queue.cc:260] T 00000000000000000000000000000000 P cf7a3184e51948d7865e7f0756fc510a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf7a3184e51948d7865e7f0756fc510a" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 36483 } }
I20260620 09:28:32.744292 28841 raft_consensus.cc:388] T 00000000000000000000000000000000 P cf7a3184e51948d7865e7f0756fc510a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:28:32.744349 28841 raft_consensus.cc:482] T 00000000000000000000000000000000 P cf7a3184e51948d7865e7f0756fc510a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:28:32.744406 28841 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cf7a3184e51948d7865e7f0756fc510a [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:28:32.744958 28841 raft_consensus.cc:504] T 00000000000000000000000000000000 P cf7a3184e51948d7865e7f0756fc510a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf7a3184e51948d7865e7f0756fc510a" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 36483 } }
I20260620 09:28:32.745085 28841 leader_election.cc:302] T 00000000000000000000000000000000 P cf7a3184e51948d7865e7f0756fc510a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cf7a3184e51948d7865e7f0756fc510a; no voters:
I20260620 09:28:32.745253 28841 leader_election.cc:288] T 00000000000000000000000000000000 P cf7a3184e51948d7865e7f0756fc510a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:28:32.745301 28846 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cf7a3184e51948d7865e7f0756fc510a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:28:32.745486 28846 raft_consensus.cc:686] T 00000000000000000000000000000000 P cf7a3184e51948d7865e7f0756fc510a [term 1 LEADER]: Becoming Leader. State: Replica: cf7a3184e51948d7865e7f0756fc510a, State: Running, Role: LEADER
I20260620 09:28:32.745596 28846 consensus_queue.cc:237] T 00000000000000000000000000000000 P cf7a3184e51948d7865e7f0756fc510a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf7a3184e51948d7865e7f0756fc510a" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 36483 } }
I20260620 09:28:32.745710 28841 sys_catalog.cc:564] T 00000000000000000000000000000000 P cf7a3184e51948d7865e7f0756fc510a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:28:32.745929 28846 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf7a3184e51948d7865e7f0756fc510a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cf7a3184e51948d7865e7f0756fc510a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf7a3184e51948d7865e7f0756fc510a" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 36483 } } }
I20260620 09:28:32.746040 28846 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf7a3184e51948d7865e7f0756fc510a [sys.catalog]: This master's current role is: LEADER
I20260620 09:28:32.746176 28846 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf7a3184e51948d7865e7f0756fc510a [sys.catalog]: SysCatalogTable state changed. Reason: New leader cf7a3184e51948d7865e7f0756fc510a. Latest consensus state: current_term: 1 leader_uuid: "cf7a3184e51948d7865e7f0756fc510a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf7a3184e51948d7865e7f0756fc510a" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 36483 } } }
I20260620 09:28:32.746265 28846 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf7a3184e51948d7865e7f0756fc510a [sys.catalog]: This master's current role is: LEADER
I20260620 09:28:32.747306 28841 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
Jun 20 09:28:32 dist-test-slave-706q krb5kdc[28329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947712, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.43.190@KRBTEST.COM
I20260620 09:28:32.749797 28850 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:28:32.751140 28850 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:28:32.754164 28850 catalog_manager.cc:1347] Generated new cluster ID: 16b7cda7e9d64aa9889667208a38004c
I20260620 09:28:32.754312 28850 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:28:32.755430 28844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:32.740408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44880 (local address 127.16.43.190:36483)
0620 09:28:32.740609 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 09:28:32.740619 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 09:28:32.740650 (+ 31us) server_negotiation.cc:407] Connection header received
0620 09:28:32.741226 (+ 576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:28:32.741236 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:28:32.741330 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:28:32.741498 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:28:32.742243 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:28:32.744140 (+ 1897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:28:32.745224 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:28:32.745847 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:28:32.749159 (+ 3312us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:28:32.749727 (+ 568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:28:32.749733 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:28:32.749733 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:28:32.749748 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:28:32.752925 (+ 3177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:28:32.753421 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:28:32.753425 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:28:32.753425 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:28:32.753547 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:28:32.753799 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:28:32.753800 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:28:32.753800 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:28:32.754192 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:28:32.754777 (+ 585us) server_negotiation.cc:1035] Waiting for connection context
0620 09:28:32.754862 (+ 85us) server_negotiation.cc:299] Negotiation successful
0620 09:28:32.755156 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":80,"threads_started":1}
I20260620 09:28:32.771275 28850 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:28:32.772655 28850 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:28:32.777891 28850 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cf7a3184e51948d7865e7f0756fc510a: Generated new TSK 0
I20260620 09:28:32.778201 28850 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.43.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.43.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.43.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.129.keytab.
Entry for principal kudu/127.16.43.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.129.keytab.
I20260620 09:28:33.486657 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.43.148:34337/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.43.129:0
--local_ip_for_outbound_sockets=127.16.43.129
--webserver_interface=127.16.43.129
--webserver_port=0
--tserver_master_addrs=127.16.43.190:36483
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.129.keytab
--principal=kudu/127.16.43.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.43.148:34845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:28:33.588454 28886 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:28:33.588761 28886 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:28:33.588799 28886 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:28:33.590516 28886 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:28:33.590580 28886 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:28:33.590621 28886 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.43.129
I20260620 09:28:33.592538 28886 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:34845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.16.43.148:34337/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.129.keytab
--principal=kudu/127.16.43.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.43.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.43.129
--webserver_port=0
--tserver_master_addrs=127.16.43.190:36483
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28886
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.43.129
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:28:33.592803 28886 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:28:33.593070 28886 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:28:33.599197 28891 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:28:33.599316 28894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:28:33.599193 28892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:28:33.600009 28886 server_base.cc:1034] running on GCE node
I20260620 09:28:33.600234 28886 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:28:33.600499 28886 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:28:33.601732 28886 hybrid_clock.cc:648] HybridClock initialized: now 1781947713601712 us; error 33 us; skew 500 ppm
Jun 20 09:28:33 dist-test-slave-706q krb5kdc[28329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947713, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:28:33.605157 28886 init.cc:377] Logged in from keytab as kudu/127.16.43.129@KRBTEST.COM (short username kudu)
I20260620 09:28:33.605808 28886 webserver.cc:460] Webserver started at http://127.16.43.129:39253/ using document root <none> and password file <none>
I20260620 09:28:33.606062 28886 fs_manager.cc:359] Metadata directory not provided
I20260620 09:28:33.606133 28886 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:28:33.606288 28886 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 09:28:33 dist-test-slave-706q krb5kdc[28329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947713, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.129@KRBTEST.COM for HTTP/127.16.43.148@KRBTEST.COM
I20260620 09:28:33.622325 28886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/ts-0/data/instance:
uuid: "98724e8ae3074eb8a38db6cffcff0fd5"
format_stamp: "Formatted at 2026-06-20 09:28:33 on dist-test-slave-706q"
server_key: "46bc46b0748460366dfff45f2614e5dd"
server_key_iv: "6cf3997fd4364fe615535135de510131"
server_key_version: "kuduclusterkey@0"
I20260620 09:28:33.622809 28886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/ts-0/wal/instance:
uuid: "98724e8ae3074eb8a38db6cffcff0fd5"
format_stamp: "Formatted at 2026-06-20 09:28:33 on dist-test-slave-706q"
server_key: "46bc46b0748460366dfff45f2614e5dd"
server_key_iv: "6cf3997fd4364fe615535135de510131"
server_key_version: "kuduclusterkey@0"
I20260620 09:28:33.625152 28886 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260620 09:28:33.639906 28901 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:28:33.640144 28886 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:28:33.640254 28886 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/ts-0/wal
uuid: "98724e8ae3074eb8a38db6cffcff0fd5"
format_stamp: "Formatted at 2026-06-20 09:28:33 on dist-test-slave-706q"
server_key: "46bc46b0748460366dfff45f2614e5dd"
server_key_iv: "6cf3997fd4364fe615535135de510131"
server_key_version: "kuduclusterkey@0"
I20260620 09:28:33.640367 28886 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:28:33.662027 28886 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:28:33.664654 28886 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:28:33.664815 28886 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:28:33.665158 28886 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:28:33.665558 28886 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:28:33.665609 28886 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:28:33.665649 28886 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:28:33.665673 28886 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:28:33.676810 28886 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.129:43613
I20260620 09:28:33.677294 28886 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/ts-0/data/info.pb
I20260620 09:28:33.678140 29016 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.129:43613 every 8 connection(s)
I20260620 09:28:33.685315 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 28886
I20260620 09:28:33.685424 16558 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/ts-0/wal/instance
I20260620 09:28:33.685616 16558 external_mini_cluster.cc:1428] Setting key 6c966c9a5eae4a1c47d5de750c3ecff7
Jun 20 09:28:33 dist-test-slave-706q krb5kdc[28329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947713, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.129@KRBTEST.COM for kudu/127.16.43.190@KRBTEST.COM
I20260620 09:28:33.692667 29019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:33.678826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:36483 (local address 127.16.43.129:34833)
0620 09:28:33.679219 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0620 09:28:33.679282 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0620 09:28:33.680014 (+ 732us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:28:33.680505 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:28:33.680510 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:28:33.680770 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:28:33.681056 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:28:33.681063 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:28:33.682105 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:28:33.682106 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:28:33.683386 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:28:33.683389 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:28:33.684004 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:28:33.684005 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:28:33.684149 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:28:33.684773 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:28:33.684786 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:28:33.688838 (+ 4052us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:28:33.690994 (+ 2156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:28:33.691005 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:28:33.691007 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:28:33.691315 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:28:33.691568 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:28:33.691569 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:28:33.691570 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:28:33.691710 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:28:33.691982 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:28:33.691988 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:28:33.692161 (+ 173us) client_negotiation.cc:769] Sending connection context
0620 09:28:33.692259 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 09:28:33.692416 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":82,"threads_started":1}
I20260620 09:28:33.693025 29020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:33.678872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.129:34833 (local address 127.16.43.190:36483)
0620 09:28:33.680233 (+ 1361us) server_negotiation.cc:206] Beginning negotiation
0620 09:28:33.680236 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:28:33.680244 (+ 8us) server_negotiation.cc:407] Connection header received
0620 09:28:33.680272 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:28:33.680273 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:28:33.680288 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:28:33.680370 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:28:33.681191 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:28:33.681990 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:28:33.683504 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:28:33.683852 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:28:33.683919 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:28:33.688961 (+ 5042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:28:33.688962 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:28:33.688962 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:28:33.688971 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:28:33.690895 (+ 1924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:28:33.691398 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:28:33.691398 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:28:33.691399 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:28:33.691460 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:28:33.691774 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:28:33.691775 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:28:33.691776 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:28:33.691907 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:28:33.691959 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 09:28:33.692841 (+ 882us) server_negotiation.cc:299] Negotiation successful
0620 09:28:33.692932 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1327,"thread_start_us":93,"threads_started":1}
I20260620 09:28:33.693536 29017 heartbeater.cc:347] Connected to a master server at 127.16.43.190:36483
I20260620 09:28:33.693672 29017 heartbeater.cc:464] Registering TS with master...
I20260620 09:28:33.693900 29017 heartbeater.cc:511] Master 127.16.43.190:36483 requested a full tablet report, sending...
I20260620 09:28:33.694481 28805 ts_manager.cc:194] Registered new tserver with Master: 98724e8ae3074eb8a38db6cffcff0fd5 (127.16.43.129:43613)
I20260620 09:28:33.695420 28805 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.43.129@KRBTEST.COM'} at 127.16.43.129:34833
WARNING: no policy specified for kudu/127.16.43.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.43.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.43.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.130.keytab.
Entry for principal kudu/127.16.43.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.130.keytab.
I20260620 09:28:33.739037 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.43.148:34337/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.43.130:0
--local_ip_for_outbound_sockets=127.16.43.130
--webserver_interface=127.16.43.130
--webserver_port=0
--tserver_master_addrs=127.16.43.190:36483
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.130.keytab
--principal=kudu/127.16.43.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.43.148:34845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 09:28:33 dist-test-slave-706q krb5kdc[28329](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781947713, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260620 09:28:33.873361 29023 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:28:33.873661 29023 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:28:33.873703 29023 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:28:33.875598 29023 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:28:33.875665 29023 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:28:33.875710 29023 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.43.130
I20260620 09:28:33.877993 29023 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:34845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.16.43.148:34337/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.130.keytab
--principal=kudu/127.16.43.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.43.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.43.130
--webserver_port=0
--tserver_master_addrs=127.16.43.190:36483
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29023
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.43.130
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:28:33.878255 29023 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:28:33.878539 29023 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:28:33.885658 29031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:28:33.885687 29033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:28:33.885895 29030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:28:33.886179 29023 server_base.cc:1034] running on GCE node
I20260620 09:28:33.886420 29023 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:28:33.886680 29023 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:28:33.887825 29023 hybrid_clock.cc:648] HybridClock initialized: now 1781947713887804 us; error 35 us; skew 500 ppm
Jun 20 09:28:33 dist-test-slave-706q krb5kdc[28329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947713, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:28:33.891105 29023 init.cc:377] Logged in from keytab as kudu/127.16.43.130@KRBTEST.COM (short username kudu)
I20260620 09:28:33.891705 29023 webserver.cc:460] Webserver started at http://127.16.43.130:33493/ using document root <none> and password file <none>
I20260620 09:28:33.891963 29023 fs_manager.cc:359] Metadata directory not provided
I20260620 09:28:33.892026 29023 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:28:33.892160 29023 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 09:28:33 dist-test-slave-706q krb5kdc[28329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947713, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.130@KRBTEST.COM for HTTP/127.16.43.148@KRBTEST.COM
I20260620 09:28:33.905409 29023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/ts-1/data/instance:
uuid: "513b49eefd0b45e0bad75656d4c8217a"
format_stamp: "Formatted at 2026-06-20 09:28:33 on dist-test-slave-706q"
server_key: "f288d8ab83d39d57573b72cf543019ca"
server_key_iv: "4e8c3216da0847c7237eb698a86f0704"
server_key_version: "kuduclusterkey@0"
I20260620 09:28:33.905866 29023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/ts-1/wal/instance:
uuid: "513b49eefd0b45e0bad75656d4c8217a"
format_stamp: "Formatted at 2026-06-20 09:28:33 on dist-test-slave-706q"
server_key: "f288d8ab83d39d57573b72cf543019ca"
server_key_iv: "4e8c3216da0847c7237eb698a86f0704"
server_key_version: "kuduclusterkey@0"
I20260620 09:28:33.908622 29023 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 09:28:33.926700 29041 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:28:33.926882 29023 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:28:33.926975 29023 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/ts-1/wal
uuid: "513b49eefd0b45e0bad75656d4c8217a"
format_stamp: "Formatted at 2026-06-20 09:28:33 on dist-test-slave-706q"
server_key: "f288d8ab83d39d57573b72cf543019ca"
server_key_iv: "4e8c3216da0847c7237eb698a86f0704"
server_key_version: "kuduclusterkey@0"
I20260620 09:28:33.927107 29023 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:28:33.941609 29023 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:28:33.943992 29023 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:28:33.944214 29023 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:28:33.944574 29023 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:28:33.944969 29023 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:28:33.945071 29023 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:28:33.945115 29023 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:28:33.945144 29023 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:28:33.952836 29023 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.130:35919
I20260620 09:28:33.952953 29154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.130:35919 every 8 connection(s)
I20260620 09:28:33.953306 29023 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/ts-1/data/info.pb
I20260620 09:28:33.954447 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 29023
I20260620 09:28:33.954556 16558 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/ts-1/wal/instance
I20260620 09:28:33.954737 16558 external_mini_cluster.cc:1428] Setting key d8a2f281a9f9b77d7d1158e57e1a33e0
Jun 20 09:28:33 dist-test-slave-706q krb5kdc[28329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947713, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.130@KRBTEST.COM for kudu/127.16.43.190@KRBTEST.COM
I20260620 09:28:33.963748 29020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:33.954205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.130:45515 (local address 127.16.43.190:36483)
0620 09:28:33.954272 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0620 09:28:33.954275 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:28:33.954557 (+ 282us) server_negotiation.cc:407] Connection header received
0620 09:28:33.955292 (+ 735us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:28:33.955294 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:28:33.955310 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:28:33.955342 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:28:33.956259 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:28:33.957015 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:28:33.958048 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:28:33.958268 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:28:33.958365 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:28:33.960796 (+ 2431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:28:33.960798 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:28:33.960798 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:28:33.960805 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:28:33.962273 (+ 1468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:28:33.962724 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:28:33.962725 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:28:33.962725 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:28:33.962785 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:28:33.963050 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:28:33.963050 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:28:33.963051 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:28:33.963174 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:28:33.963253 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 09:28:33.963565 (+ 312us) server_negotiation.cc:299] Negotiation successful
0620 09:28:33.963655 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 09:28:33.964025 29157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:33.954153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:36483 (local address 127.16.43.130:45515)
0620 09:28:33.954411 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0620 09:28:33.954468 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 09:28:33.955159 (+ 691us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:28:33.955588 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:28:33.955593 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:28:33.955853 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:28:33.956146 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:28:33.956154 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:28:33.957118 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:28:33.957119 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:28:33.957938 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:28:33.957941 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:28:33.958389 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:28:33.958390 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:28:33.958514 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:28:33.959085 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:28:33.959094 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:28:33.960676 (+ 1582us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:28:33.962382 (+ 1706us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:28:33.962388 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:28:33.962391 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:28:33.962628 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:28:33.962853 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:28:33.962854 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:28:33.962855 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:28:33.962979 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:28:33.963243 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:28:33.963249 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:28:33.963405 (+ 156us) client_negotiation.cc:769] Sending connection context
0620 09:28:33.963519 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 09:28:33.963697 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":81,"threads_started":1}
WARNING: no policy specified for kudu/127.16.43.131@KRBTEST.COM; defaulting to no policy
I20260620 09:28:33.964519 29155 heartbeater.cc:347] Connected to a master server at 127.16.43.190:36483
I20260620 09:28:33.964633 29155 heartbeater.cc:464] Registering TS with master...
I20260620 09:28:33.964828 29155 heartbeater.cc:511] Master 127.16.43.190:36483 requested a full tablet report, sending...
I20260620 09:28:33.965225 28805 ts_manager.cc:194] Registered new tserver with Master: 513b49eefd0b45e0bad75656d4c8217a (127.16.43.130:35919)
I20260620 09:28:33.965607 28805 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.43.130@KRBTEST.COM'} at 127.16.43.130:45515
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.43.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.43.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.131.keytab.
Entry for principal kudu/127.16.43.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.131.keytab.
I20260620 09:28:33.987920 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.43.148:34337/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.43.131:0
--local_ip_for_outbound_sockets=127.16.43.131
--webserver_interface=127.16.43.131
--webserver_port=0
--tserver_master_addrs=127.16.43.190:36483
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.131.keytab
--principal=kudu/127.16.43.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.43.148:34845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:28:34.111230 29160 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:28:34.111548 29160 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:28:34.111595 29160 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:28:34.112840 29160 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:28:34.112890 29160 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:28:34.112926 29160 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.43.131
I20260620 09:28:34.114975 29160 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:34845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.16.43.148:34337/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.131.keytab
--principal=kudu/127.16.43.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.43.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.43.131
--webserver_port=0
--tserver_master_addrs=127.16.43.190:36483
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29160
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.43.131
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:28:34.115236 29160 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:28:34.115506 29160 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:28:34.120708 29166 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:28:34.120906 29160 server_base.cc:1034] running on GCE node
W20260620 09:28:34.120736 29165 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:28:34.120967 29168 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:28:34.121187 29160 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:28:34.121450 29160 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:28:34.122627 29160 hybrid_clock.cc:648] HybridClock initialized: now 1781947714122613 us; error 32 us; skew 500 ppm
Jun 20 09:28:34 dist-test-slave-706q krb5kdc[28329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947714, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:28:34.125262 29160 init.cc:377] Logged in from keytab as kudu/127.16.43.131@KRBTEST.COM (short username kudu)
I20260620 09:28:34.125716 29160 webserver.cc:460] Webserver started at http://127.16.43.131:34959/ using document root <none> and password file <none>
I20260620 09:28:34.125973 29160 fs_manager.cc:359] Metadata directory not provided
I20260620 09:28:34.126044 29160 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:28:34.126192 29160 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 09:28:34 dist-test-slave-706q krb5kdc[28329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947714, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.131@KRBTEST.COM for HTTP/127.16.43.148@KRBTEST.COM
I20260620 09:28:34.139878 29160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/ts-2/data/instance:
uuid: "de541ced876949c0a20ac957353d8942"
format_stamp: "Formatted at 2026-06-20 09:28:34 on dist-test-slave-706q"
server_key: "8687e1af2d294204d37398714f2e5e9a"
server_key_iv: "7c107aeaf7346c6dbe9898b79a815ab6"
server_key_version: "kuduclusterkey@0"
I20260620 09:28:34.140429 29160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/ts-2/wal/instance:
uuid: "de541ced876949c0a20ac957353d8942"
format_stamp: "Formatted at 2026-06-20 09:28:34 on dist-test-slave-706q"
server_key: "8687e1af2d294204d37398714f2e5e9a"
server_key_iv: "7c107aeaf7346c6dbe9898b79a815ab6"
server_key_version: "kuduclusterkey@0"
I20260620 09:28:34.144719 29160 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260620 09:28:34.159612 29175 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:28:34.159853 29160 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 09:28:34.159963 29160 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/ts-2/wal
uuid: "de541ced876949c0a20ac957353d8942"
format_stamp: "Formatted at 2026-06-20 09:28:34 on dist-test-slave-706q"
server_key: "8687e1af2d294204d37398714f2e5e9a"
server_key_iv: "7c107aeaf7346c6dbe9898b79a815ab6"
server_key_version: "kuduclusterkey@0"
I20260620 09:28:34.160080 29160 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:28:34.186156 29160 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:28:34.189005 29160 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:28:34.189205 29160 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:28:34.189503 29160 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:28:34.189949 29160 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:28:34.190012 29160 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:28:34.190063 29160 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:28:34.190096 29160 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:28:34.200093 29160 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.131:41575
I20260620 09:28:34.200644 29160 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/ts-2/data/info.pb
I20260620 09:28:34.206454 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 29160
I20260620 09:28:34.206570 16558 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/ts-2/wal/instance
I20260620 09:28:34.206785 16558 external_mini_cluster.cc:1428] Setting key acadcb850703682ef959b25b650474b0
Jun 20 09:28:34 dist-test-slave-706q krb5kdc[28329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947714, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.131@KRBTEST.COM for kudu/127.16.43.190@KRBTEST.COM
I20260620 09:28:34.210273 29288 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.131:41575 every 8 connection(s)
I20260620 09:28:34.213251 29020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:34.201770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.131:54447 (local address 127.16.43.190:36483)
0620 09:28:34.201817 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 09:28:34.201820 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:28:34.202028 (+ 208us) server_negotiation.cc:407] Connection header received
0620 09:28:34.202810 (+ 782us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:28:34.202812 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:28:34.202824 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:28:34.202858 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:28:34.203783 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:28:34.204563 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:28:34.206148 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:28:34.206440 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:28:34.206498 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:28:34.209421 (+ 2923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:28:34.209423 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:28:34.209423 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:28:34.209432 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:28:34.211460 (+ 2028us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:28:34.212024 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:28:34.212025 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:28:34.212025 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:28:34.212089 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:28:34.212409 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:28:34.212410 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:28:34.212411 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:28:34.212583 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:28:34.212674 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 09:28:34.213041 (+ 367us) server_negotiation.cc:299] Negotiation successful
0620 09:28:34.213145 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 09:28:34.213497 29291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:34.201484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:36483 (local address 127.16.43.131:54447)
0620 09:28:34.201894 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0620 09:28:34.201942 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 09:28:34.202676 (+ 734us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:28:34.203057 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:28:34.203061 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:28:34.203327 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:28:34.203668 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:28:34.203675 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:28:34.204685 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:28:34.204686 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:28:34.206046 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:28:34.206048 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:28:34.206652 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:28:34.206653 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:28:34.206801 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:28:34.207632 (+ 831us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:28:34.207643 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:28:34.209312 (+ 1669us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:28:34.211605 (+ 2293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:28:34.211612 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:28:34.211615 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:28:34.211924 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:28:34.212167 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:28:34.212167 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:28:34.212168 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:28:34.212305 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:28:34.212683 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:28:34.212689 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:28:34.212863 (+ 174us) client_negotiation.cc:769] Sending connection context
0620 09:28:34.212993 (+ 130us) client_negotiation.cc:240] Negotiation successful
0620 09:28:34.213166 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":81,"threads_started":1}
I20260620 09:28:34.213937 29289 heartbeater.cc:347] Connected to a master server at 127.16.43.190:36483
I20260620 09:28:34.214043 29289 heartbeater.cc:464] Registering TS with master...
I20260620 09:28:34.214251 29289 heartbeater.cc:511] Master 127.16.43.190:36483 requested a full tablet report, sending...
I20260620 09:28:34.214669 28805 ts_manager.cc:194] Registered new tserver with Master: de541ced876949c0a20ac957353d8942 (127.16.43.131:41575)
I20260620 09:28:34.215183 28805 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.43.131@KRBTEST.COM'} at 127.16.43.131:54447
I20260620 09:28:34.219480 16558 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 09:28:34.220304 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 28886
I20260620 09:28:34.228232 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 29023
I20260620 09:28:34.233982 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 29160
I20260620 09:28:34.241071 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 28773
I20260620 09:28:34.246197 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.43.190:36483
--webserver_interface=127.16.43.190
--webserver_port=33705
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.190.keytab
--principal=kudu/127.16.43.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.43.148:34845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.43.190:36483
--ranger_config_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.43.148:34337/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.16.43.148:34337/kms with env {KRB5CCNAME=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:28:34.333134 29292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:28:34.333356 29292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:28:34.333380 29292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:28:34.334511 29292 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:28:34.334548 29292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:28:34.334563 29292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:28:34.334575 29292 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:28:34.334587 29292 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:28:34.336225 29292 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:34845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.16.43.148:34337/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.16.43.190:36483
--ranger_config_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.190.keytab
--principal=kudu/127.16.43.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.43.190:36483
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.43.190
--webserver_port=33705
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:28:34.336691 29292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:28:34.336968 29292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:28:34.342799 29298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:28:34.342823 29297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:28:34.342942 29292 server_base.cc:1034] running on GCE node
W20260620 09:28:34.342877 29300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:28:34.343354 29292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:28:34.343690 29292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:28:34.344864 29292 hybrid_clock.cc:648] HybridClock initialized: now 1781947714344852 us; error 42 us; skew 500 ppm
Jun 20 09:28:34 dist-test-slave-706q krb5kdc[28329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947714, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:28:34.347395 29292 init.cc:377] Logged in from keytab as kudu/127.16.43.190@KRBTEST.COM (short username kudu)
I20260620 09:28:34.347886 29292 webserver.cc:460] Webserver started at http://127.16.43.190:33705/ using document root <none> and password file <none>
I20260620 09:28:34.348145 29292 fs_manager.cc:359] Metadata directory not provided
I20260620 09:28:34.348222 29292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 09:28:34 dist-test-slave-706q krb5kdc[28329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947714, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.190@KRBTEST.COM for HTTP/127.16.43.148@KRBTEST.COM
I20260620 09:28:34.392287 29292 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:28:34.393352 29308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:28:34.393626 29292 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:28:34.393744 29292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/master-0/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/master-0/wal
uuid: "cf7a3184e51948d7865e7f0756fc510a"
format_stamp: "Formatted at 2026-06-20 09:28:32 on dist-test-slave-706q"
server_key: "7c4542e727ca8934624a1050d9f5b50e"
server_key_iv: "d55969e6284df3de3136e7a71e9741d9"
server_key_version: "kuduclusterkey@0"
I20260620 09:28:34.394292 29292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:28:34.419242 29292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:28:34.421123 29292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:28:34.421242 29292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:28:34.425457 29292 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.190:36483
I20260620 09:28:34.425504 29360 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.190:36483 every 8 connection(s)
I20260620 09:28:34.425868 29292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
I20260620 09:28:34.427156 29361 sys_catalog.cc:263] Verifying existing consensus state
I20260620 09:28:34.427887 29361 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf7a3184e51948d7865e7f0756fc510a: Bootstrap starting.
I20260620 09:28:34.434456 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 29292
I20260620 09:28:34.434842 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.43.148:34337/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.43.129:43613
--local_ip_for_outbound_sockets=127.16.43.129
--tserver_master_addrs=127.16.43.190:36483
--webserver_port=39253
--webserver_interface=127.16.43.129
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.129.keytab
--principal=kudu/127.16.43.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.43.148:34845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.16.43.148:34337/kms with env {KRB5CCNAME=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 09:28:34.436579 29361 log.cc:824] T 00000000000000000000000000000000 P cf7a3184e51948d7865e7f0756fc510a: Log is configured to *not* fsync() on all Append() calls
I20260620 09:28:34.438458 29361 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf7a3184e51948d7865e7f0756fc510a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 09:28:34.438757 29361 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf7a3184e51948d7865e7f0756fc510a: Bootstrap complete.
I20260620 09:28:34.441305 29361 raft_consensus.cc:348] T 00000000000000000000000000000000 P cf7a3184e51948d7865e7f0756fc510a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf7a3184e51948d7865e7f0756fc510a" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 36483 } }
I20260620 09:28:34.441551 29361 raft_consensus.cc:729] T 00000000000000000000000000000000 P cf7a3184e51948d7865e7f0756fc510a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf7a3184e51948d7865e7f0756fc510a, State: Initialized, Role: FOLLOWER
I20260620 09:28:34.441725 29361 consensus_queue.cc:260] T 00000000000000000000000000000000 P cf7a3184e51948d7865e7f0756fc510a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf7a3184e51948d7865e7f0756fc510a" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 36483 } }
I20260620 09:28:34.441797 29361 raft_consensus.cc:388] T 00000000000000000000000000000000 P cf7a3184e51948d7865e7f0756fc510a [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:28:34.441870 29361 raft_consensus.cc:482] T 00000000000000000000000000000000 P cf7a3184e51948d7865e7f0756fc510a [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:28:34.441913 29361 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cf7a3184e51948d7865e7f0756fc510a [term 1 FOLLOWER]: Advancing to term 2
I20260620 09:28:34.442970 29361 raft_consensus.cc:504] T 00000000000000000000000000000000 P cf7a3184e51948d7865e7f0756fc510a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf7a3184e51948d7865e7f0756fc510a" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 36483 } }
I20260620 09:28:34.443060 29361 leader_election.cc:302] T 00000000000000000000000000000000 P cf7a3184e51948d7865e7f0756fc510a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cf7a3184e51948d7865e7f0756fc510a; no voters:
I20260620 09:28:34.443291 29361 leader_election.cc:288] T 00000000000000000000000000000000 P cf7a3184e51948d7865e7f0756fc510a [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 09:28:34.443336 29365 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cf7a3184e51948d7865e7f0756fc510a [term 2 FOLLOWER]: Leader election won for term 2
I20260620 09:28:34.443465 29365 raft_consensus.cc:686] T 00000000000000000000000000000000 P cf7a3184e51948d7865e7f0756fc510a [term 2 LEADER]: Becoming Leader. State: Replica: cf7a3184e51948d7865e7f0756fc510a, State: Running, Role: LEADER
I20260620 09:28:34.443593 29361 sys_catalog.cc:564] T 00000000000000000000000000000000 P cf7a3184e51948d7865e7f0756fc510a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:28:34.443791 29365 consensus_queue.cc:237] T 00000000000000000000000000000000 P cf7a3184e51948d7865e7f0756fc510a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf7a3184e51948d7865e7f0756fc510a" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 36483 } }
I20260620 09:28:34.444096 29365 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf7a3184e51948d7865e7f0756fc510a [sys.catalog]: SysCatalogTable state changed. Reason: New leader cf7a3184e51948d7865e7f0756fc510a. Latest consensus state: current_term: 2 leader_uuid: "cf7a3184e51948d7865e7f0756fc510a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf7a3184e51948d7865e7f0756fc510a" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 36483 } } }
I20260620 09:28:34.444195 29365 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf7a3184e51948d7865e7f0756fc510a [sys.catalog]: This master's current role is: LEADER
I20260620 09:28:34.444103 29366 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf7a3184e51948d7865e7f0756fc510a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "cf7a3184e51948d7865e7f0756fc510a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf7a3184e51948d7865e7f0756fc510a" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 36483 } } }
I20260620 09:28:34.444381 29366 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf7a3184e51948d7865e7f0756fc510a [sys.catalog]: This master's current role is: LEADER
I20260620 09:28:34.444929 29374 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:28:34.445329 29361 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 09:28:34.445400 29374 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:28:34.445525 29374 catalog_manager.cc:1259] Loaded cluster ID: 16b7cda7e9d64aa9889667208a38004c
I20260620 09:28:34.445559 29374 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:28:34.449353 29374 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:28:34.449913 29374 catalog_manager.cc:5879] T 00000000000000000000000000000000 P cf7a3184e51948d7865e7f0756fc510a: Loaded TSK: 0
I20260620 09:28:34.450201 29374 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 09:28:34.555670 29363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:28:34.555954 29363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:28:34.556005 29363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:28:34.557857 29363 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:28:34.557924 29363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:28:34.557966 29363 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.43.129
I20260620 09:28:34.560352 29363 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:34845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.16.43.148:34337/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.129.keytab
--principal=kudu/127.16.43.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.43.129:43613
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.43.129
--webserver_port=39253
--tserver_master_addrs=127.16.43.190:36483
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29363
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.43.129
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:28:34.560604 29363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:28:34.560910 29363 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:28:34.567833 29399 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:28:34.568113 29398 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:28:34.568396 29401 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:28:34.569196 29363 server_base.cc:1034] running on GCE node
I20260620 09:28:34.569480 29363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:28:34.569782 29363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:28:34.570960 29363 hybrid_clock.cc:648] HybridClock initialized: now 1781947714570936 us; error 37 us; skew 500 ppm
Jun 20 09:28:34 dist-test-slave-706q krb5kdc[28329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947714, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:28:34.574236 29363 init.cc:377] Logged in from keytab as kudu/127.16.43.129@KRBTEST.COM (short username kudu)
I20260620 09:28:34.574787 29363 webserver.cc:460] Webserver started at http://127.16.43.129:39253/ using document root <none> and password file <none>
I20260620 09:28:34.575069 29363 fs_manager.cc:359] Metadata directory not provided
I20260620 09:28:34.575222 29363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 09:28:34 dist-test-slave-706q krb5kdc[28329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947714, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.129@KRBTEST.COM for HTTP/127.16.43.148@KRBTEST.COM
I20260620 09:28:34.594785 29363 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:28:34.596306 29409 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:28:34.596498 29363 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:28:34.596585 29363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/ts-0/wal
uuid: "98724e8ae3074eb8a38db6cffcff0fd5"
format_stamp: "Formatted at 2026-06-20 09:28:33 on dist-test-slave-706q"
server_key: "46bc46b0748460366dfff45f2614e5dd"
server_key_iv: "6cf3997fd4364fe615535135de510131"
server_key_version: "kuduclusterkey@0"
I20260620 09:28:34.597087 29363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:28:34.613956 29363 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:28:34.616550 29363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:28:34.616755 29363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:28:34.617031 29363 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:28:34.617393 29363 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:28:34.617455 29363 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:28:34.617504 29363 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:28:34.617539 29363 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:28:34.624678 29363 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.129:43613
I20260620 09:28:34.624766 29522 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.129:43613 every 8 connection(s)
I20260620 09:28:34.625222 29363 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/ts-0/data/info.pb
I20260620 09:28:34.631011 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 29363
I20260620 09:28:34.631431 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.43.148:34337/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.43.130:35919
--local_ip_for_outbound_sockets=127.16.43.130
--tserver_master_addrs=127.16.43.190:36483
--webserver_port=33493
--webserver_interface=127.16.43.130
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.130.keytab
--principal=kudu/127.16.43.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.43.148:34845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.16.43.148:34337/kms with env {KRB5CCNAME=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 09:28:34 dist-test-slave-706q krb5kdc[28329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947714, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.129@KRBTEST.COM for kudu/127.16.43.190@KRBTEST.COM
I20260620 09:28:34.639338 29526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:34.626156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.129:33673 (local address 127.16.43.190:36483)
0620 09:28:34.626379 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0620 09:28:34.626387 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 09:28:34.626426 (+ 39us) server_negotiation.cc:407] Connection header received
0620 09:28:34.627109 (+ 683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:28:34.627114 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:28:34.627167 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:28:34.627316 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:28:34.627986 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:28:34.628909 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:28:34.630032 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:28:34.630700 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:28:34.630855 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:28:34.633833 (+ 2978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:28:34.633838 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:28:34.633838 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:28:34.633852 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:28:34.636132 (+ 2280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:28:34.636690 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:28:34.636694 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:28:34.636695 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:28:34.636822 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:28:34.638191 (+ 1369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:28:34.638192 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:28:34.638192 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:28:34.638522 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:28:34.638588 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 09:28:34.639000 (+ 412us) server_negotiation.cc:299] Negotiation successful
0620 09:28:34.639117 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":84,"threads_started":1}
I20260620 09:28:34.639349 29525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:34.626057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:36483 (local address 127.16.43.129:33673)
0620 09:28:34.626236 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 09:28:34.626298 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0620 09:28:34.626829 (+ 531us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:28:34.627354 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:28:34.627358 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:28:34.627652 (+ 294us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:28:34.627856 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:28:34.627862 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:28:34.629054 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:28:34.629056 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:28:34.629888 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:28:34.629890 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:28:34.630890 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:28:34.630892 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:28:34.631047 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:28:34.631788 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:28:34.631806 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:28:34.633655 (+ 1849us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:28:34.636253 (+ 2598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:28:34.636265 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:28:34.636267 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:28:34.636580 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:28:34.636915 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:28:34.636916 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:28:34.636916 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:28:34.637055 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:28:34.638647 (+ 1592us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:28:34.638653 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:28:34.638797 (+ 144us) client_negotiation.cc:769] Sending connection context
0620 09:28:34.638921 (+ 124us) client_negotiation.cc:240] Negotiation successful
0620 09:28:34.639106 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":90,"threads_started":1}
I20260620 09:28:34.640736 29523 heartbeater.cc:347] Connected to a master server at 127.16.43.190:36483
I20260620 09:28:34.640851 29523 heartbeater.cc:464] Registering TS with master...
I20260620 09:28:34.641075 29523 heartbeater.cc:511] Master 127.16.43.190:36483 requested a full tablet report, sending...
W20260620 09:28:34.641970 29523 heartbeater.cc:644] Failed to heartbeat to 127.16.43.190:36483 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 09:28:34.642046 29523 heartbeater.cc:464] Registering TS with master...
I20260620 09:28:34.642109 29523 heartbeater.cc:511] Master 127.16.43.190:36483 requested a full tablet report, sending...
I20260620 09:28:34.642463 29523 heartbeater.cc:464] Registering TS with master...
I20260620 09:28:34.642547 29523 heartbeater.cc:511] Master 127.16.43.190:36483 requested a full tablet report, sending...
W20260620 09:28:34.642935 29523 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 09:28:34.737514 29527 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:28:34.737784 29527 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:28:34.737838 29527 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:28:34.739919 29527 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:28:34.739977 29527 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:28:34.740026 29527 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.43.130
I20260620 09:28:34.742460 29527 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:34845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.16.43.148:34337/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.130.keytab
--principal=kudu/127.16.43.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.43.130:35919
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.43.130
--webserver_port=33493
--tserver_master_addrs=127.16.43.190:36483
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29527
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.43.130
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:28:34.742764 29527 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:28:34.743062 29527 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:28:34.749758 29535 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:28:34.749753 29533 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:28:34.750850 29527 server_base.cc:1034] running on GCE node
W20260620 09:28:34.751579 29532 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:28:34.751845 29527 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:28:34.752141 29527 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:28:34.753352 29527 hybrid_clock.cc:648] HybridClock initialized: now 1781947714753291 us; error 74 us; skew 500 ppm
Jun 20 09:28:34 dist-test-slave-706q krb5kdc[28329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947714, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:28:34.756501 29527 init.cc:377] Logged in from keytab as kudu/127.16.43.130@KRBTEST.COM (short username kudu)
I20260620 09:28:34.757089 29527 webserver.cc:460] Webserver started at http://127.16.43.130:33493/ using document root <none> and password file <none>
I20260620 09:28:34.757431 29527 fs_manager.cc:359] Metadata directory not provided
I20260620 09:28:34.757524 29527 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 09:28:34 dist-test-slave-706q krb5kdc[28329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947714, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.130@KRBTEST.COM for HTTP/127.16.43.148@KRBTEST.COM
I20260620 09:28:34.776263 29527 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.000s
I20260620 09:28:34.781122 29543 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:28:34.781399 29527 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.000s
I20260620 09:28:34.781509 29527 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/ts-1/wal
uuid: "513b49eefd0b45e0bad75656d4c8217a"
format_stamp: "Formatted at 2026-06-20 09:28:33 on dist-test-slave-706q"
server_key: "f288d8ab83d39d57573b72cf543019ca"
server_key_iv: "4e8c3216da0847c7237eb698a86f0704"
server_key_version: "kuduclusterkey@0"
I20260620 09:28:34.782028 29527 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:28:34.804796 29527 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:28:34.807183 29527 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:28:34.807369 29527 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:28:34.807673 29527 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:28:34.808073 29527 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:28:34.808122 29527 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:28:34.808161 29527 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:28:34.808183 29527 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:28:34.815371 29527 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.130:35919
I20260620 09:28:34.815965 29527 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/ts-1/data/info.pb
I20260620 09:28:34.816612 29656 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.130:35919 every 8 connection(s)
Jun 20 09:28:34 dist-test-slave-706q krb5kdc[28329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947714, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.130@KRBTEST.COM for kudu/127.16.43.190@KRBTEST.COM
I20260620 09:28:34.822566 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 29527
I20260620 09:28:34.822842 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.43.148:34337/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.43.131:41575
--local_ip_for_outbound_sockets=127.16.43.131
--tserver_master_addrs=127.16.43.190:36483
--webserver_port=34959
--webserver_interface=127.16.43.131
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.131.keytab
--principal=kudu/127.16.43.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.43.148:34845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.16.43.148:34337/kms with env {KRB5CCNAME=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 09:28:34.828512 29526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:34.817125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.130:37467 (local address 127.16.43.190:36483)
0620 09:28:34.817200 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0620 09:28:34.817202 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:28:34.817215 (+ 13us) server_negotiation.cc:407] Connection header received
0620 09:28:34.817778 (+ 563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:28:34.817780 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:28:34.817793 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:28:34.817869 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:28:34.818602 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:28:34.819181 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:28:34.820374 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:28:34.820592 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:28:34.820681 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:28:34.824661 (+ 3980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:28:34.824663 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:28:34.824664 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:28:34.824674 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:28:34.826624 (+ 1950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:28:34.827207 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:28:34.827209 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:28:34.827209 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:28:34.827276 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:28:34.827648 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:28:34.827650 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:28:34.827650 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:28:34.827812 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:28:34.827891 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 09:28:34.828306 (+ 415us) server_negotiation.cc:299] Negotiation successful
0620 09:28:34.828410 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 09:28:34.829090 29659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:34.816768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:36483 (local address 127.16.43.130:37467)
0620 09:28:34.816948 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0620 09:28:34.816994 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 09:28:34.817631 (+ 637us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:28:34.817949 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:28:34.817953 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:28:34.818178 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:28:34.818478 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:28:34.818485 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:28:34.819290 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:28:34.819291 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:28:34.820275 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:28:34.820277 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:28:34.820713 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:28:34.820715 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:28:34.820844 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:28:34.821438 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:28:34.821447 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:28:34.824532 (+ 3085us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:28:34.826712 (+ 2180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:28:34.826721 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:28:34.826724 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:28:34.827113 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:28:34.827372 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:28:34.827373 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:28:34.827374 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:28:34.827574 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:28:34.827893 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:28:34.827899 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:28:34.828108 (+ 209us) client_negotiation.cc:769] Sending connection context
0620 09:28:34.828235 (+ 127us) client_negotiation.cc:240] Negotiation successful
0620 09:28:34.828426 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":109,"threads_started":1}
I20260620 09:28:34.829813 29657 heartbeater.cc:347] Connected to a master server at 127.16.43.190:36483
I20260620 09:28:34.829914 29657 heartbeater.cc:464] Registering TS with master...
I20260620 09:28:34.830153 29657 heartbeater.cc:511] Master 127.16.43.190:36483 requested a full tablet report, sending...
W20260620 09:28:34.830873 29657 heartbeater.cc:644] Failed to heartbeat to 127.16.43.190:36483 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 09:28:34.830942 29657 heartbeater.cc:464] Registering TS with master...
I20260620 09:28:34.831028 29657 heartbeater.cc:511] Master 127.16.43.190:36483 requested a full tablet report, sending...
I20260620 09:28:34.831387 29657 heartbeater.cc:464] Registering TS with master...
I20260620 09:28:34.831468 29657 heartbeater.cc:511] Master 127.16.43.190:36483 requested a full tablet report, sending...
W20260620 09:28:34.831835 29657 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 09:28:34.925675 29660 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:28:34.925910 29660 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:28:34.925953 29660 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:28:34.927729 29660 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:28:34.927790 29660 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:28:34.927836 29660 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.43.131
I20260620 09:28:34.929329 29660 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:34845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.16.43.148:34337/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.131.keytab
--principal=kudu/127.16.43.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.43.131:41575
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.43.131
--webserver_port=34959
--tserver_master_addrs=127.16.43.190:36483
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29660
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.43.131
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:28:34.929615 29660 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:28:34.929868 29660 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:28:34.935283 29668 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:28:34.935643 29665 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:28:34.935277 29666 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:28:34.936007 29660 server_base.cc:1034] running on GCE node
I20260620 09:28:34.936215 29660 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:28:34.936491 29660 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:28:34.937659 29660 hybrid_clock.cc:648] HybridClock initialized: now 1781947714937637 us; error 36 us; skew 500 ppm
Jun 20 09:28:34 dist-test-slave-706q krb5kdc[28329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947714, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:28:34.940243 29660 init.cc:377] Logged in from keytab as kudu/127.16.43.131@KRBTEST.COM (short username kudu)
I20260620 09:28:34.940802 29660 webserver.cc:460] Webserver started at http://127.16.43.131:34959/ using document root <none> and password file <none>
I20260620 09:28:34.941018 29660 fs_manager.cc:359] Metadata directory not provided
I20260620 09:28:34.941063 29660 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 09:28:34 dist-test-slave-706q krb5kdc[28329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947714, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.131@KRBTEST.COM for HTTP/127.16.43.148@KRBTEST.COM
I20260620 09:28:34.958563 29660 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:28:34.959671 29677 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:28:34.959903 29660 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:28:34.960005 29660 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/ts-2/wal
uuid: "de541ced876949c0a20ac957353d8942"
format_stamp: "Formatted at 2026-06-20 09:28:34 on dist-test-slave-706q"
server_key: "8687e1af2d294204d37398714f2e5e9a"
server_key_iv: "7c107aeaf7346c6dbe9898b79a815ab6"
server_key_version: "kuduclusterkey@0"
I20260620 09:28:34.960561 29660 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:28:34.980508 29660 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:28:34.982725 29660 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:28:34.982849 29660 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:28:34.983083 29660 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:28:34.983455 29660 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:28:34.983495 29660 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:28:34.983572 29660 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:28:34.983603 29660 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:28:34.990741 29660 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.131:41575
I20260620 09:28:34.990778 29790 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.131:41575 every 8 connection(s)
I20260620 09:28:34.991220 29660 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/ts-2/data/info.pb
I20260620 09:28:34.997953 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 29660
Jun 20 09:28:34 dist-test-slave-706q krb5kdc[28329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947714, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.131@KRBTEST.COM for kudu/127.16.43.190@KRBTEST.COM
I20260620 09:28:35.004243 29526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:34.994207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.131:49325 (local address 127.16.43.190:36483)
0620 09:28:34.994259 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 09:28:34.994261 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:28:34.994515 (+ 254us) server_negotiation.cc:407] Connection header received
0620 09:28:34.995033 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:28:34.995034 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:28:34.995046 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:28:34.995103 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:28:34.995749 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:28:34.996259 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:28:34.997730 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:28:34.998034 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:28:34.998082 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:28:35.000795 (+ 2713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:28:35.000797 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:28:35.000798 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:28:35.000805 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:28:35.002580 (+ 1775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:28:35.003077 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:28:35.003079 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:28:35.003079 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:28:35.003140 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:28:35.003428 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:28:35.003429 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:28:35.003429 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:28:35.003619 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:28:35.003668 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 09:28:35.004048 (+ 380us) server_negotiation.cc:299] Negotiation successful
0620 09:28:35.004139 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 09:28:35.005028 29793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:34.994219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:36483 (local address 127.16.43.131:49325)
0620 09:28:34.994377 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 09:28:34.994419 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 09:28:34.994921 (+ 502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:28:34.995178 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:28:34.995182 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:28:34.995365 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:28:34.995635 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:28:34.995641 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:28:34.996349 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:28:34.996350 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:28:34.997619 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:28:34.997623 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:28:34.998193 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:28:34.998194 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:28:34.998539 (+ 345us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:28:34.999177 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:28:34.999187 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:28:35.000679 (+ 1492us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:28:35.002677 (+ 1998us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:28:35.002684 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:28:35.002686 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:28:35.002982 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:28:35.003213 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:28:35.003213 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:28:35.003214 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:28:35.003346 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:28:35.003719 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:28:35.003726 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:28:35.003890 (+ 164us) client_negotiation.cc:769] Sending connection context
0620 09:28:35.004617 (+ 727us) client_negotiation.cc:240] Negotiation successful
0620 09:28:35.004790 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":90,"threads_started":1}
I20260620 09:28:35.005741 29791 heartbeater.cc:347] Connected to a master server at 127.16.43.190:36483
I20260620 09:28:35.005868 29791 heartbeater.cc:464] Registering TS with master...
I20260620 09:28:35.006057 29791 heartbeater.cc:511] Master 127.16.43.190:36483 requested a full tablet report, sending...
W20260620 09:28:35.006726 29791 heartbeater.cc:644] Failed to heartbeat to 127.16.43.190:36483 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 09:28:35.006806 29791 heartbeater.cc:464] Registering TS with master...
I20260620 09:28:35.006879 29791 heartbeater.cc:511] Master 127.16.43.190:36483 requested a full tablet report, sending...
I20260620 09:28:35.007185 29791 heartbeater.cc:464] Registering TS with master...
I20260620 09:28:35.007259 29791 heartbeater.cc:511] Master 127.16.43.190:36483 requested a full tablet report, sending...
W20260620 09:28:35.007551 29791 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 09:28:35.014222 29526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:34.998421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44886 (local address 127.16.43.190:36483)
0620 09:28:35.004320 (+ 5899us) server_negotiation.cc:206] Beginning negotiation
0620 09:28:35.004322 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:28:35.007775 (+ 3453us) server_negotiation.cc:407] Connection header received
0620 09:28:35.007861 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:28:35.007862 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:28:35.007874 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:28:35.007928 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:28:35.008518 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:28:35.008992 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:28:35.009768 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:28:35.009977 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:28:35.010067 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:28:35.011343 (+ 1276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:28:35.011345 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:28:35.011345 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:28:35.011353 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:28:35.012979 (+ 1626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:28:35.013363 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:28:35.013364 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:28:35.013365 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:28:35.013419 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:28:35.013631 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:28:35.013632 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:28:35.013633 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:28:35.013766 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:28:35.013964 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0620 09:28:35.014016 (+ 52us) server_negotiation.cc:299] Negotiation successful
0620 09:28:35.014105 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5884,"thread_start_us":62,"threads_started":1}
Jun 20 09:28:35 dist-test-slave-706q krb5kdc[28329](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781947715, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:28:35.643177 29523 heartbeater.cc:464] Registering TS with master...
I20260620 09:28:35.643342 29523 heartbeater.cc:511] Master 127.16.43.190:36483 requested a full tablet report, sending...
I20260620 09:28:35.643790 29325 ts_manager.cc:194] Registered new tserver with Master: 98724e8ae3074eb8a38db6cffcff0fd5 (127.16.43.129:43613)
I20260620 09:28:35.644860 29325 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.43.129@KRBTEST.COM'} at 127.16.43.129:33673
I20260620 09:28:35.832008 29657 heartbeater.cc:464] Registering TS with master...
I20260620 09:28:35.832172 29657 heartbeater.cc:511] Master 127.16.43.190:36483 requested a full tablet report, sending...
I20260620 09:28:35.832440 29325 ts_manager.cc:194] Registered new tserver with Master: 513b49eefd0b45e0bad75656d4c8217a (127.16.43.130:35919)
I20260620 09:28:35.832844 29325 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.43.130@KRBTEST.COM'} at 127.16.43.130:37467
Jun 20 09:28:35 dist-test-slave-706q krb5kdc[28329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947715, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.190@KRBTEST.COM for HTTP/127.16.43.148@KRBTEST.COM
I20260620 09:28:36.007719 29791 heartbeater.cc:464] Registering TS with master...
I20260620 09:28:36.007893 29791 heartbeater.cc:511] Master 127.16.43.190:36483 requested a full tablet report, sending...
I20260620 09:28:36.008222 29325 ts_manager.cc:194] Registered new tserver with Master: de541ced876949c0a20ac957353d8942 (127.16.43.131:41575)
I20260620 09:28:36.008711 29325 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.43.131@KRBTEST.COM'} at 127.16.43.131:49325
I20260620 09:28:36.014968 16558 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 09:28:36.023875 29813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:36.015992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44892 (local address 127.16.43.190:36483)
0620 09:28:36.016121 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 09:28:36.016123 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:28:36.016133 (+ 10us) server_negotiation.cc:407] Connection header received
0620 09:28:36.016201 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:28:36.016201 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:28:36.016213 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:28:36.016278 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:28:36.016944 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:28:36.017751 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:28:36.018696 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:28:36.019020 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:28:36.019089 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:28:36.020617 (+ 1528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:28:36.020619 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:28:36.020619 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:28:36.020627 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:28:36.022461 (+ 1834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:28:36.022857 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:28:36.022858 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:28:36.022859 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:28:36.022921 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:28:36.023130 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:28:36.023131 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:28:36.023131 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:28:36.023275 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:28:36.023371 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 09:28:36.023521 (+ 150us) server_negotiation.cc:299] Negotiation successful
0620 09:28:36.023637 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":78,"threads_started":1}
I20260620 09:28:36.026006 29325 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44892:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 09:28:36.028422 29325 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 09:28:36.036465 29822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:36.031453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.129:43613 (local address 127.0.0.1:51992)
0620 09:28:36.032046 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0620 09:28:36.032085 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 09:28:36.032159 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:28:36.032380 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:28:36.032383 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:28:36.032385 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:28:36.032457 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:28:36.032458 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:28:36.033678 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:28:36.033679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:28:36.035012 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:28:36.035015 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:28:36.036115 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:28:36.036116 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:28:36.036211 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:28:36.036229 (+ 18us) client_negotiation.cc:769] Sending connection context
0620 09:28:36.036283 (+ 54us) client_negotiation.cc:240] Negotiation successful
0620 09:28:36.036345 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":197,"threads_started":1}
I20260620 09:28:36.037348 29821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:36.031708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51992 (local address 127.16.43.129:43613)
0620 09:28:36.031984 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0620 09:28:36.031989 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:28:36.032163 (+ 174us) server_negotiation.cc:407] Connection header received
0620 09:28:36.032239 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:28:36.032241 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:28:36.032258 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:28:36.032370 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:28:36.032517 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:28:36.033589 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:28:36.035114 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:28:36.035991 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:28:36.037078 (+ 1087us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:28:36.037170 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 09:28:36.037203 (+ 33us) server_negotiation.cc:299] Negotiation successful
0620 09:28:36.037256 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238,"thread_start_us":70,"threads_started":1}
I20260620 09:28:36.038324 29457 tablet_service.cc:1467] Processing CreateTablet for tablet cc9038ca57764c53a25c715ffa4c5204 (DEFAULT_TABLE table=test-table [id=47fe6aa0291c4e6cb9148a1d00a3e942]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:28:36.038759 29457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc9038ca57764c53a25c715ffa4c5204. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:28:36.043828 29822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:36.031778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.131:41575 (local address 127.0.0.1:33474)
0620 09:28:36.036553 (+ 4775us) negotiation.cc:107] Waiting for socket to connect
0620 09:28:36.036564 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 09:28:36.036614 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:28:36.037899 (+ 1285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:28:36.037900 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:28:36.037902 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:28:36.037935 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:28:36.037937 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:28:36.039404 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:28:36.039405 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:28:36.040661 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:28:36.040664 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:28:36.043618 (+ 2954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:28:36.043619 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:28:36.043678 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:28:36.043682 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 09:28:36.043710 (+ 28us) client_negotiation.cc:240] Negotiation successful
0620 09:28:36.043758 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4764,"thread_start_us":94,"threads_started":1}
I20260620 09:28:36.044627 29823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:36.031889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33474 (local address 127.16.43.131:41575)
0620 09:28:36.032292 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0620 09:28:36.032297 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:28:36.037727 (+ 5430us) server_negotiation.cc:407] Connection header received
0620 09:28:36.037773 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:28:36.037776 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:28:36.037795 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:28:36.038006 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:28:36.038092 (+ 86us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:28:36.039290 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:28:36.040773 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:28:36.041612 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:28:36.044272 (+ 2660us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:28:36.044379 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 09:28:36.044499 (+ 120us) server_negotiation.cc:299] Negotiation successful
0620 09:28:36.044547 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":54,"threads_started":1}
I20260620 09:28:36.044770 29818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:36.031466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.130:35919 (local address 127.0.0.1:51458)
0620 09:28:36.035624 (+ 4158us) negotiation.cc:107] Waiting for socket to connect
0620 09:28:36.035635 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 09:28:36.035684 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:28:36.036041 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:28:36.036042 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:28:36.036044 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:28:36.036084 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:28:36.036087 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:28:36.041780 (+ 5693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:28:36.041781 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:28:36.043053 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:28:36.043056 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:28:36.044513 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:28:36.044514 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:28:36.044584 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:28:36.044588 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 09:28:36.044640 (+ 52us) client_negotiation.cc:240] Negotiation successful
0620 09:28:36.044683 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4139,"thread_start_us":108,"threads_started":1}
I20260620 09:28:36.045275 29819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:36.031583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51458 (local address 127.16.43.130:35919)
0620 09:28:36.035802 (+ 4219us) server_negotiation.cc:206] Beginning negotiation
0620 09:28:36.035807 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:28:36.035817 (+ 10us) server_negotiation.cc:407] Connection header received
0620 09:28:36.035862 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:28:36.035864 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:28:36.035880 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:28:36.035958 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:28:36.036154 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:28:36.041693 (+ 5539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:28:36.043350 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:28:36.044335 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:28:36.044417 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:28:36.044477 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 09:28:36.045139 (+ 662us) server_negotiation.cc:299] Negotiation successful
0620 09:28:36.045183 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4184,"thread_start_us":53,"threads_started":1}
I20260620 09:28:36.045964 29591 tablet_service.cc:1467] Processing CreateTablet for tablet cc9038ca57764c53a25c715ffa4c5204 (DEFAULT_TABLE table=test-table [id=47fe6aa0291c4e6cb9148a1d00a3e942]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:28:36.046309 29591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc9038ca57764c53a25c715ffa4c5204. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:28:36.050143 29825 tablet_bootstrap.cc:492] T cc9038ca57764c53a25c715ffa4c5204 P 513b49eefd0b45e0bad75656d4c8217a: Bootstrap starting.
I20260620 09:28:36.050889 29825 tablet_bootstrap.cc:654] T cc9038ca57764c53a25c715ffa4c5204 P 513b49eefd0b45e0bad75656d4c8217a: Neither blocks nor log segments found. Creating new log.
I20260620 09:28:36.051206 29825 log.cc:824] T cc9038ca57764c53a25c715ffa4c5204 P 513b49eefd0b45e0bad75656d4c8217a: Log is configured to *not* fsync() on all Append() calls
I20260620 09:28:36.052285 29722 tablet_service.cc:1467] Processing CreateTablet for tablet cc9038ca57764c53a25c715ffa4c5204 (DEFAULT_TABLE table=test-table [id=47fe6aa0291c4e6cb9148a1d00a3e942]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:28:36.052397 29825 tablet_bootstrap.cc:492] T cc9038ca57764c53a25c715ffa4c5204 P 513b49eefd0b45e0bad75656d4c8217a: No bootstrap required, opened a new log
I20260620 09:28:36.052461 29825 ts_tablet_manager.cc:1397] T cc9038ca57764c53a25c715ffa4c5204 P 513b49eefd0b45e0bad75656d4c8217a: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260620 09:28:36.052629 29722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc9038ca57764c53a25c715ffa4c5204. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:28:36.052654 29824 tablet_bootstrap.cc:492] T cc9038ca57764c53a25c715ffa4c5204 P 98724e8ae3074eb8a38db6cffcff0fd5: Bootstrap starting.
I20260620 09:28:36.053426 29824 tablet_bootstrap.cc:654] T cc9038ca57764c53a25c715ffa4c5204 P 98724e8ae3074eb8a38db6cffcff0fd5: Neither blocks nor log segments found. Creating new log.
I20260620 09:28:36.053745 29824 log.cc:824] T cc9038ca57764c53a25c715ffa4c5204 P 98724e8ae3074eb8a38db6cffcff0fd5: Log is configured to *not* fsync() on all Append() calls
I20260620 09:28:36.055464 29828 tablet_bootstrap.cc:492] T cc9038ca57764c53a25c715ffa4c5204 P de541ced876949c0a20ac957353d8942: Bootstrap starting.
I20260620 09:28:36.056260 29824 tablet_bootstrap.cc:492] T cc9038ca57764c53a25c715ffa4c5204 P 98724e8ae3074eb8a38db6cffcff0fd5: No bootstrap required, opened a new log
I20260620 09:28:36.056280 29828 tablet_bootstrap.cc:654] T cc9038ca57764c53a25c715ffa4c5204 P de541ced876949c0a20ac957353d8942: Neither blocks nor log segments found. Creating new log.
I20260620 09:28:36.056409 29824 ts_tablet_manager.cc:1397] T cc9038ca57764c53a25c715ffa4c5204 P 98724e8ae3074eb8a38db6cffcff0fd5: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260620 09:28:36.056574 29828 log.cc:824] T cc9038ca57764c53a25c715ffa4c5204 P de541ced876949c0a20ac957353d8942: Log is configured to *not* fsync() on all Append() calls
I20260620 09:28:36.059130 29824 raft_consensus.cc:348] T cc9038ca57764c53a25c715ffa4c5204 P 98724e8ae3074eb8a38db6cffcff0fd5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98724e8ae3074eb8a38db6cffcff0fd5" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 43613 } } peers { permanent_uuid: "de541ced876949c0a20ac957353d8942" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 41575 } } peers { permanent_uuid: "513b49eefd0b45e0bad75656d4c8217a" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 35919 } }
I20260620 09:28:36.060541 29825 raft_consensus.cc:348] T cc9038ca57764c53a25c715ffa4c5204 P 513b49eefd0b45e0bad75656d4c8217a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98724e8ae3074eb8a38db6cffcff0fd5" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 43613 } } peers { permanent_uuid: "de541ced876949c0a20ac957353d8942" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 41575 } } peers { permanent_uuid: "513b49eefd0b45e0bad75656d4c8217a" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 35919 } }
I20260620 09:28:36.060729 29825 raft_consensus.cc:374] T cc9038ca57764c53a25c715ffa4c5204 P 513b49eefd0b45e0bad75656d4c8217a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:28:36.060762 29825 raft_consensus.cc:729] T cc9038ca57764c53a25c715ffa4c5204 P 513b49eefd0b45e0bad75656d4c8217a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 513b49eefd0b45e0bad75656d4c8217a, State: Initialized, Role: FOLLOWER
I20260620 09:28:36.061029 29825 consensus_queue.cc:260] T cc9038ca57764c53a25c715ffa4c5204 P 513b49eefd0b45e0bad75656d4c8217a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98724e8ae3074eb8a38db6cffcff0fd5" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 43613 } } peers { permanent_uuid: "de541ced876949c0a20ac957353d8942" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 41575 } } peers { permanent_uuid: "513b49eefd0b45e0bad75656d4c8217a" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 35919 } }
I20260620 09:28:36.061430 29825 ts_tablet_manager.cc:1428] T cc9038ca57764c53a25c715ffa4c5204 P 513b49eefd0b45e0bad75656d4c8217a: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20260620 09:28:36.061456 29828 tablet_bootstrap.cc:492] T cc9038ca57764c53a25c715ffa4c5204 P de541ced876949c0a20ac957353d8942: No bootstrap required, opened a new log
I20260620 09:28:36.061524 29828 ts_tablet_manager.cc:1397] T cc9038ca57764c53a25c715ffa4c5204 P de541ced876949c0a20ac957353d8942: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260620 09:28:36.061653 29657 heartbeater.cc:503] Master 127.16.43.190:36483 was elected leader, sending a full tablet report...
I20260620 09:28:36.062681 29824 raft_consensus.cc:374] T cc9038ca57764c53a25c715ffa4c5204 P 98724e8ae3074eb8a38db6cffcff0fd5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:28:36.062812 29824 raft_consensus.cc:729] T cc9038ca57764c53a25c715ffa4c5204 P 98724e8ae3074eb8a38db6cffcff0fd5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98724e8ae3074eb8a38db6cffcff0fd5, State: Initialized, Role: FOLLOWER
I20260620 09:28:36.063035 29824 consensus_queue.cc:260] T cc9038ca57764c53a25c715ffa4c5204 P 98724e8ae3074eb8a38db6cffcff0fd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98724e8ae3074eb8a38db6cffcff0fd5" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 43613 } } peers { permanent_uuid: "de541ced876949c0a20ac957353d8942" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 41575 } } peers { permanent_uuid: "513b49eefd0b45e0bad75656d4c8217a" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 35919 } }
I20260620 09:28:36.063397 29824 ts_tablet_manager.cc:1428] T cc9038ca57764c53a25c715ffa4c5204 P 98724e8ae3074eb8a38db6cffcff0fd5: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20260620 09:28:36.063586 29523 heartbeater.cc:503] Master 127.16.43.190:36483 was elected leader, sending a full tablet report...
I20260620 09:28:36.063737 29828 raft_consensus.cc:348] T cc9038ca57764c53a25c715ffa4c5204 P de541ced876949c0a20ac957353d8942 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98724e8ae3074eb8a38db6cffcff0fd5" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 43613 } } peers { permanent_uuid: "de541ced876949c0a20ac957353d8942" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 41575 } } peers { permanent_uuid: "513b49eefd0b45e0bad75656d4c8217a" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 35919 } }
I20260620 09:28:36.063910 29828 raft_consensus.cc:374] T cc9038ca57764c53a25c715ffa4c5204 P de541ced876949c0a20ac957353d8942 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:28:36.063962 29828 raft_consensus.cc:729] T cc9038ca57764c53a25c715ffa4c5204 P de541ced876949c0a20ac957353d8942 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de541ced876949c0a20ac957353d8942, State: Initialized, Role: FOLLOWER
I20260620 09:28:36.064093 29828 consensus_queue.cc:260] T cc9038ca57764c53a25c715ffa4c5204 P de541ced876949c0a20ac957353d8942 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98724e8ae3074eb8a38db6cffcff0fd5" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 43613 } } peers { permanent_uuid: "de541ced876949c0a20ac957353d8942" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 41575 } } peers { permanent_uuid: "513b49eefd0b45e0bad75656d4c8217a" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 35919 } }
I20260620 09:28:36.064347 29828 ts_tablet_manager.cc:1428] T cc9038ca57764c53a25c715ffa4c5204 P de541ced876949c0a20ac957353d8942: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260620 09:28:36.064487 29791 heartbeater.cc:503] Master 127.16.43.190:36483 was elected leader, sending a full tablet report...
W20260620 09:28:36.067579 29658 tablet.cc:2369] T cc9038ca57764c53a25c715ffa4c5204 P 513b49eefd0b45e0bad75656d4c8217a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 09:28:36.127007 29524 tablet.cc:2369] T cc9038ca57764c53a25c715ffa4c5204 P 98724e8ae3074eb8a38db6cffcff0fd5: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 09:28:36 dist-test-slave-706q krb5kdc[28329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947715, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.190@KRBTEST.COM for HTTP/127.16.43.148@KRBTEST.COM
I20260620 09:28:36.197768 29832 raft_consensus.cc:482] T cc9038ca57764c53a25c715ffa4c5204 P de541ced876949c0a20ac957353d8942 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:28:36.197928 29832 raft_consensus.cc:504] T cc9038ca57764c53a25c715ffa4c5204 P de541ced876949c0a20ac957353d8942 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98724e8ae3074eb8a38db6cffcff0fd5" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 43613 } } peers { permanent_uuid: "de541ced876949c0a20ac957353d8942" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 41575 } } peers { permanent_uuid: "513b49eefd0b45e0bad75656d4c8217a" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 35919 } }
I20260620 09:28:36.198344 29832 leader_election.cc:288] T cc9038ca57764c53a25c715ffa4c5204 P de541ced876949c0a20ac957353d8942 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98724e8ae3074eb8a38db6cffcff0fd5 (127.16.43.129:43613), 513b49eefd0b45e0bad75656d4c8217a (127.16.43.130:35919)
I20260620 09:28:36.212914 29821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:36.199406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.131:38321 (local address 127.16.43.129:43613)
0620 09:28:36.199469 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0620 09:28:36.199472 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:28:36.203165 (+ 3693us) server_negotiation.cc:407] Connection header received
0620 09:28:36.203601 (+ 436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:28:36.203603 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:28:36.203614 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:28:36.203644 (+ 30us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:28:36.205417 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:28:36.206164 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:28:36.210369 (+ 4205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:28:36.210917 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:28:36.210968 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:28:36.210978 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 09:28:36.212774 (+ 1796us) server_negotiation.cc:299] Negotiation successful
0620 09:28:36.212834 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260620 09:28:36.213785 29834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:36.198826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.129:43613 (local address 127.16.43.131:38321)
0620 09:28:36.203096 (+ 4270us) negotiation.cc:107] Waiting for socket to connect
0620 09:28:36.203111 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 09:28:36.203418 (+ 307us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:28:36.204101 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:28:36.204102 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:28:36.204105 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:28:36.204339 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:28:36.204343 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:28:36.206414 (+ 2071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:28:36.206415 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:28:36.209799 (+ 3384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:28:36.209805 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:28:36.211212 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:28:36.211213 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:28:36.211283 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:28:36.211287 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 09:28:36.213623 (+ 2336us) client_negotiation.cc:240] Negotiation successful
0620 09:28:36.213677 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4244,"thread_start_us":104,"threads_started":1}
I20260620 09:28:36.214466 29477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc9038ca57764c53a25c715ffa4c5204" candidate_uuid: "de541ced876949c0a20ac957353d8942" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98724e8ae3074eb8a38db6cffcff0fd5" is_pre_election: true
I20260620 09:28:36.214646 29477 raft_consensus.cc:2454] T cc9038ca57764c53a25c715ffa4c5204 P 98724e8ae3074eb8a38db6cffcff0fd5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de541ced876949c0a20ac957353d8942 in term 0.
I20260620 09:28:36.214851 29819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:36.199488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.131:47439 (local address 127.16.43.130:35919)
0620 09:28:36.199549 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 09:28:36.199551 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:28:36.209753 (+ 10202us) server_negotiation.cc:407] Connection header received
0620 09:28:36.209876 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:28:36.209878 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:28:36.209890 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:28:36.209923 (+ 33us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:28:36.210153 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:28:36.210803 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:28:36.212093 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:28:36.212618 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:28:36.212661 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:28:36.212668 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0620 09:28:36.214727 (+ 2059us) server_negotiation.cc:299] Negotiation successful
0620 09:28:36.214784 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 09:28:36.214890 29680 leader_election.cc:302] T cc9038ca57764c53a25c715ffa4c5204 P de541ced876949c0a20ac957353d8942 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 98724e8ae3074eb8a38db6cffcff0fd5, de541ced876949c0a20ac957353d8942; no voters:
I20260620 09:28:36.215065 29832 raft_consensus.cc:2781] T cc9038ca57764c53a25c715ffa4c5204 P de541ced876949c0a20ac957353d8942 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 09:28:36.215142 29832 raft_consensus.cc:482] T cc9038ca57764c53a25c715ffa4c5204 P de541ced876949c0a20ac957353d8942 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 09:28:36.215163 29832 raft_consensus.cc:3037] T cc9038ca57764c53a25c715ffa4c5204 P de541ced876949c0a20ac957353d8942 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:28:36.215307 29835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:36.199077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.130:35919 (local address 127.16.43.131:47439)
0620 09:28:36.209664 (+ 10587us) negotiation.cc:107] Waiting for socket to connect
0620 09:28:36.209679 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 09:28:36.209790 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:28:36.209996 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:28:36.209997 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:28:36.209999 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:28:36.210071 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:28:36.210073 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:28:36.210914 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:28:36.210915 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:28:36.211978 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:28:36.211981 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:28:36.212744 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:28:36.212745 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:28:36.212826 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:28:36.212830 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 09:28:36.215170 (+ 2340us) client_negotiation.cc:240] Negotiation successful
0620 09:28:36.215220 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10553,"thread_start_us":79,"threads_started":1}
I20260620 09:28:36.215912 29601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc9038ca57764c53a25c715ffa4c5204" candidate_uuid: "de541ced876949c0a20ac957353d8942" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "513b49eefd0b45e0bad75656d4c8217a" is_pre_election: true
I20260620 09:28:36.216087 29601 raft_consensus.cc:2454] T cc9038ca57764c53a25c715ffa4c5204 P 513b49eefd0b45e0bad75656d4c8217a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de541ced876949c0a20ac957353d8942 in term 0.
I20260620 09:28:36.216113 29832 raft_consensus.cc:504] T cc9038ca57764c53a25c715ffa4c5204 P de541ced876949c0a20ac957353d8942 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98724e8ae3074eb8a38db6cffcff0fd5" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 43613 } } peers { permanent_uuid: "de541ced876949c0a20ac957353d8942" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 41575 } } peers { permanent_uuid: "513b49eefd0b45e0bad75656d4c8217a" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 35919 } }
I20260620 09:28:36.216244 29832 leader_election.cc:288] T cc9038ca57764c53a25c715ffa4c5204 P de541ced876949c0a20ac957353d8942 [CANDIDATE]: Term 1 election: Requested vote from peers 98724e8ae3074eb8a38db6cffcff0fd5 (127.16.43.129:43613), 513b49eefd0b45e0bad75656d4c8217a (127.16.43.130:35919)
I20260620 09:28:36.216581 29601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc9038ca57764c53a25c715ffa4c5204" candidate_uuid: "de541ced876949c0a20ac957353d8942" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "513b49eefd0b45e0bad75656d4c8217a"
I20260620 09:28:36.216576 29477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc9038ca57764c53a25c715ffa4c5204" candidate_uuid: "de541ced876949c0a20ac957353d8942" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98724e8ae3074eb8a38db6cffcff0fd5"
I20260620 09:28:36.216660 29601 raft_consensus.cc:3037] T cc9038ca57764c53a25c715ffa4c5204 P 513b49eefd0b45e0bad75656d4c8217a [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:28:36.216665 29477 raft_consensus.cc:3037] T cc9038ca57764c53a25c715ffa4c5204 P 98724e8ae3074eb8a38db6cffcff0fd5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:28:36.217451 29601 raft_consensus.cc:2454] T cc9038ca57764c53a25c715ffa4c5204 P 513b49eefd0b45e0bad75656d4c8217a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de541ced876949c0a20ac957353d8942 in term 1.
I20260620 09:28:36.217617 29477 raft_consensus.cc:2454] T cc9038ca57764c53a25c715ffa4c5204 P 98724e8ae3074eb8a38db6cffcff0fd5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de541ced876949c0a20ac957353d8942 in term 1.
I20260620 09:28:36.217633 29678 leader_election.cc:302] T cc9038ca57764c53a25c715ffa4c5204 P de541ced876949c0a20ac957353d8942 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 513b49eefd0b45e0bad75656d4c8217a, de541ced876949c0a20ac957353d8942; no voters:
I20260620 09:28:36.217870 29832 raft_consensus.cc:2781] T cc9038ca57764c53a25c715ffa4c5204 P de541ced876949c0a20ac957353d8942 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:28:36.218029 29832 raft_consensus.cc:686] T cc9038ca57764c53a25c715ffa4c5204 P de541ced876949c0a20ac957353d8942 [term 1 LEADER]: Becoming Leader. State: Replica: de541ced876949c0a20ac957353d8942, State: Running, Role: LEADER
I20260620 09:28:36.218125 29832 consensus_queue.cc:237] T cc9038ca57764c53a25c715ffa4c5204 P de541ced876949c0a20ac957353d8942 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98724e8ae3074eb8a38db6cffcff0fd5" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 43613 } } peers { permanent_uuid: "de541ced876949c0a20ac957353d8942" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 41575 } } peers { permanent_uuid: "513b49eefd0b45e0bad75656d4c8217a" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 35919 } }
I20260620 09:28:36.219003 29324 catalog_manager.cc:5495] T cc9038ca57764c53a25c715ffa4c5204 P de541ced876949c0a20ac957353d8942 reported cstate change: term changed from 0 to 1, leader changed from <none> to de541ced876949c0a20ac957353d8942 (127.16.43.131). New cstate: current_term: 1 leader_uuid: "de541ced876949c0a20ac957353d8942" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98724e8ae3074eb8a38db6cffcff0fd5" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 43613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "de541ced876949c0a20ac957353d8942" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 41575 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "513b49eefd0b45e0bad75656d4c8217a" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 35919 } health_report { overall_health: UNKNOWN } } }
W20260620 09:28:36.244576 29792 tablet.cc:2369] T cc9038ca57764c53a25c715ffa4c5204 P de541ced876949c0a20ac957353d8942: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 09:28:36.280714 29823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:36.272508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33480 (local address 127.16.43.131:41575)
0620 09:28:36.272560 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 09:28:36.272563 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:28:36.272573 (+ 10us) server_negotiation.cc:407] Connection header received
0620 09:28:36.272621 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:28:36.272622 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:28:36.272646 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:28:36.272736 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:28:36.272949 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:28:36.273800 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:28:36.279175 (+ 5375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:28:36.279611 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:28:36.279723 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:28:36.280005 (+ 282us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:28:36.280170 (+ 165us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:28:36.280260 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 09:28:36.280586 (+ 326us) server_negotiation.cc:299] Negotiation successful
0620 09:28:36.280616 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 09:28:36.283771 29601 raft_consensus.cc:1261] T cc9038ca57764c53a25c715ffa4c5204 P 513b49eefd0b45e0bad75656d4c8217a [term 1 FOLLOWER]: Refusing update from remote peer de541ced876949c0a20ac957353d8942: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:28:36.284298 29836 consensus_queue.cc:1035] T cc9038ca57764c53a25c715ffa4c5204 P de541ced876949c0a20ac957353d8942 [LEADER]: Connected to new peer: Peer: permanent_uuid: "513b49eefd0b45e0bad75656d4c8217a" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 35919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:28:36.289678 29477 raft_consensus.cc:1261] T cc9038ca57764c53a25c715ffa4c5204 P 98724e8ae3074eb8a38db6cffcff0fd5 [term 1 FOLLOWER]: Refusing update from remote peer de541ced876949c0a20ac957353d8942: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:28:36.290162 29836 consensus_queue.cc:1035] T cc9038ca57764c53a25c715ffa4c5204 P de541ced876949c0a20ac957353d8942 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98724e8ae3074eb8a38db6cffcff0fd5" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 43613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:28:36.295281 29840 mvcc.cc:205] Tried to move back new op lower bound from 7298857845893246976 to 7298857845629992960. Current Snapshot: MvccSnapshot[applied={T|T < 7298857845893246976}]
I20260620 09:28:36.295617 29843 mvcc.cc:205] Tried to move back new op lower bound from 7298857845893246976 to 7298857845629992960. Current Snapshot: MvccSnapshot[applied={T|T < 7298857845893246976}]
I20260620 09:28:36.297612 29841 mvcc.cc:205] Tried to move back new op lower bound from 7298857845893246976 to 7298857845629992960. Current Snapshot: MvccSnapshot[applied={T|T < 7298857845893246976}]
I20260620 09:28:36.299955 29324 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44892:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:28:36.300078 29324 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44892:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:28:36.301103 29324 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cf7a3184e51948d7865e7f0756fc510a: Sending DeleteTablet for 3 replicas of tablet cc9038ca57764c53a25c715ffa4c5204
I20260620 09:28:36.301539 29591 tablet_service.cc:1514] Processing DeleteTablet for tablet cc9038ca57764c53a25c715ffa4c5204 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:28:36 UTC) from {username='kudu', principal='kudu/127.16.43.190@KRBTEST.COM'} at 127.0.0.1:51458
I20260620 09:28:36.301554 29457 tablet_service.cc:1514] Processing DeleteTablet for tablet cc9038ca57764c53a25c715ffa4c5204 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:28:36 UTC) from {username='kudu', principal='kudu/127.16.43.190@KRBTEST.COM'} at 127.0.0.1:51992
I20260620 09:28:36.302196 29722 tablet_service.cc:1514] Processing DeleteTablet for tablet cc9038ca57764c53a25c715ffa4c5204 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:28:36 UTC) from {username='kudu', principal='kudu/127.16.43.190@KRBTEST.COM'} at 127.0.0.1:33474
I20260620 09:28:36.302750 29850 tablet_replica.cc:330] T cc9038ca57764c53a25c715ffa4c5204 P 98724e8ae3074eb8a38db6cffcff0fd5: stopping tablet replica
I20260620 09:28:36.302925 29850 raft_consensus.cc:2229] T cc9038ca57764c53a25c715ffa4c5204 P 98724e8ae3074eb8a38db6cffcff0fd5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:28:36.302929 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 29363
I20260620 09:28:36.311010 29851 tablet_replica.cc:330] T cc9038ca57764c53a25c715ffa4c5204 P de541ced876949c0a20ac957353d8942: stopping tablet replica
I20260620 09:28:36.311254 29851 raft_consensus.cc:2229] T cc9038ca57764c53a25c715ffa4c5204 P de541ced876949c0a20ac957353d8942 [term 1 LEADER]: Raft consensus shutting down.
I20260620 09:28:36.311483 29851 raft_consensus.cc:2258] T cc9038ca57764c53a25c715ffa4c5204 P de541ced876949c0a20ac957353d8942 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:28:36.311656 29849 tablet_replica.cc:330] T cc9038ca57764c53a25c715ffa4c5204 P 513b49eefd0b45e0bad75656d4c8217a: stopping tablet replica
I20260620 09:28:36.311838 29849 raft_consensus.cc:2229] T cc9038ca57764c53a25c715ffa4c5204 P 513b49eefd0b45e0bad75656d4c8217a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:28:36.311959 29849 raft_consensus.cc:2258] T cc9038ca57764c53a25c715ffa4c5204 P 513b49eefd0b45e0bad75656d4c8217a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:28:36.312199 29851 ts_tablet_manager.cc:1905] T cc9038ca57764c53a25c715ffa4c5204 P de541ced876949c0a20ac957353d8942: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 09:28:36.312270 29311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.43.129:43613 (error 108)
I20260620 09:28:36.312897 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 29527
I20260620 09:28:36.314199 29851 ts_tablet_manager.cc:1918] T cc9038ca57764c53a25c715ffa4c5204 P de541ced876949c0a20ac957353d8942: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:28:36.314275 29851 log.cc:1196] T cc9038ca57764c53a25c715ffa4c5204 P de541ced876949c0a20ac957353d8942: Deleting WAL directory at /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781947562949448-16558-0/minicluster-data/ts-2/wal/wals/cc9038ca57764c53a25c715ffa4c5204
I20260620 09:28:36.314564 29851 ts_tablet_manager.cc:1939] T cc9038ca57764c53a25c715ffa4c5204 P de541ced876949c0a20ac957353d8942: Deleting consensus metadata
I20260620 09:28:36.315311 29312 catalog_manager.cc:4841] TS de541ced876949c0a20ac957353d8942 (127.16.43.131:41575): tablet cc9038ca57764c53a25c715ffa4c5204 (table test-table [id=47fe6aa0291c4e6cb9148a1d00a3e942]) successfully deleted
I20260620 09:28:36.322809 29818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:36.322663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.129:43613 (local address 127.0.0.1:48698)
0620 09:28:36.322723 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0620 09:28:36.322764 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.129:43613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260620 09:28:36.322988 29311 catalog_manager.cc:4584] TS 98724e8ae3074eb8a38db6cffcff0fd5 (127.16.43.129:43613): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cc9038ca57764c53a25c715ffa4c5204: Network error: Client connection negotiation failed: client connection to 127.16.43.129:43613: connect: Connection refused (error 111)
I20260620 09:28:36.326833 29818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:36.326706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.130:35919 (local address 127.0.0.1:47680)
0620 09:28:36.326763 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0620 09:28:36.326794 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.130:35919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260620 09:28:36.327117 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 29660
I20260620 09:28:36.333637 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 29292
I20260620 09:28:36.341370 16558 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 09:28:36.686218 16558 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 09:28:36.686424 16558 mini_ranger.cc:66] Stopping Ranger...
I20260620 09:28:37.031177 16558 mini_ranger.cc:68] Stopped Ranger
2026-06-20 09:28:37.031 UTC [28359] LOG: received smart shutdown request
2026-06-20 09:28:37.035 UTC [28359] LOG: background worker "logical replication launcher" (PID 28368) exited with exit code 1
2026-06-20 09:28:37.035 UTC [28363] LOG: shutting down
2026-06-20 09:28:37.210 UTC [28359] LOG: database system is shut down
2026-06-20T09:28:37Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (48836 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781947562949448-16558-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:28:37 dist-test-slave-706q krb5kdc[29861](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:28:37 dist-test-slave-706q krb5kdc[29861](info): set up 2 sockets
Jun 20 09:28:37 dist-test-slave-706q krb5kdc[29861](info): commencing operation
krb5kdc: starting...
W20260620 09:28:38.372562 16558 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781947562949448-16558-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781947562949448-16558-0/krb5kdc/test-user.keytab.
Jun 20 09:28:38 dist-test-slave-706q krb5kdc[29861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947718, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:28:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:28:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.43.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.43.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.43.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.190.keytab.
Entry for principal kudu/127.16.43.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.190.keytab.
I20260620 09:28:38.496780 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781947562949448-16558-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781947562949448-16558-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781947562949448-16558-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.43.190:40067
--webserver_interface=127.16.43.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.190.keytab
--principal=kudu/127.16.43.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.43.148:41667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.43.190:40067,127.16.43.189:45261,127.16.43.188:41383
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781947562949448-16558-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781947562949448-16558-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781947562949448-16558-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:28:38.580542 29874 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:28:38.580760 29874 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:28:38.580782 29874 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:28:38.581996 29874 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:28:38.582041 29874 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:28:38.582055 29874 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:28:38.582068 29874 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:28:38.582080 29874 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:28:38.583663 29874 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:41667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781947562949448-16558-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781947562949448-16558-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.43.190:40067,127.16.43.189:45261,127.16.43.188:41383
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.190.keytab
--principal=kudu/127.16.43.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.43.190:40067
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.43.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29874
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781947562949448-16558-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:28:38.583916 29874 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:28:38.584161 29874 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:28:38.589188 29882 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:28:38.589188 29879 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:28:38.589207 29880 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:28:38.589515 29874 server_base.cc:1034] running on GCE node
I20260620 09:28:38.589813 29874 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:28:38.590116 29874 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:28:38.591264 29874 hybrid_clock.cc:648] HybridClock initialized: now 1781947718591245 us; error 41 us; skew 500 ppm
Jun 20 09:28:38 dist-test-slave-706q krb5kdc[29861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947718, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:28:38.594029 29874 init.cc:377] Logged in from keytab as kudu/127.16.43.190@KRBTEST.COM (short username kudu)
I20260620 09:28:38.594435 29874 webserver.cc:460] Webserver started at http://127.16.43.190:41187/ using document root <none> and password file <none>
I20260620 09:28:38.594640 29874 fs_manager.cc:359] Metadata directory not provided
I20260620 09:28:38.594679 29874 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:28:38.594769 29874 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:28:38.595845 29874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781947562949448-16558-0/minicluster-data/master-0/data/instance:
uuid: "e1cebf2b7d7b40dc85e05405ed6a8369"
format_stamp: "Formatted at 2026-06-20 09:28:38 on dist-test-slave-706q"
server_key: "376f23b652b9a3814a30620fa4d9ca6a"
server_key_iv: "8db1e7da5aec2fd811f0c2065957273b"
server_key_version: "encryptionkey@0"
I20260620 09:28:38.596148 29874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781947562949448-16558-0/minicluster-data/master-0/wal/instance:
uuid: "e1cebf2b7d7b40dc85e05405ed6a8369"
format_stamp: "Formatted at 2026-06-20 09:28:38 on dist-test-slave-706q"
server_key: "376f23b652b9a3814a30620fa4d9ca6a"
server_key_iv: "8db1e7da5aec2fd811f0c2065957273b"
server_key_version: "encryptionkey@0"
I20260620 09:28:38.597492 29874 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260620 09:28:38.598181 29889 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:28:38.598356 29874 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 09:28:38.598492 29874 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781947562949448-16558-0/minicluster-data/master-0/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781947562949448-16558-0/minicluster-data/master-0/wal
uuid: "e1cebf2b7d7b40dc85e05405ed6a8369"
format_stamp: "Formatted at 2026-06-20 09:28:38 on dist-test-slave-706q"
server_key: "376f23b652b9a3814a30620fa4d9ca6a"
server_key_iv: "8db1e7da5aec2fd811f0c2065957273b"
server_key_version: "encryptionkey@0"
I20260620 09:28:38.598572 29874 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781947562949448-16558-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781947562949448-16558-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781947562949448-16558-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:28:38.615386 29874 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:28:38.617552 29874 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:28:38.617729 29874 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:28:38.621785 29874 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.190:40067
I20260620 09:28:38.621873 29941 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.190:40067 every 8 connection(s)
I20260620 09:28:38.622202 29874 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
I20260620 09:28:38.623183 29942 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:28:38.624861 29942 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.43.190" port: 40067 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 09:28:38 dist-test-slave-706q krb5kdc[29861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947718, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.190@KRBTEST.COM for kudu/127.16.43.190@KRBTEST.COM
I20260620 09:28:38.632148 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 29874
I20260620 09:28:38.632267 16558 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781947562949448-16558-0/minicluster-data/master-0/wal/instance
I20260620 09:28:38.632563 16558 external_mini_cluster.cc:1428] Setting key 1d45099c789389ab601a48258ef3e040
I20260620 09:28:38.635502 29944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:38.625564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54318 (local address 127.16.43.190:40067)
0620 09:28:38.626290 (+ 726us) server_negotiation.cc:206] Beginning negotiation
0620 09:28:38.626293 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:28:38.626303 (+ 10us) server_negotiation.cc:407] Connection header received
0620 09:28:38.626362 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:28:38.626363 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:28:38.626379 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:28:38.626489 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:28:38.627081 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:28:38.628093 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:28:38.628979 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:28:38.629322 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:28:38.629477 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:28:38.631717 (+ 2240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:28:38.631724 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:28:38.631724 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:28:38.631731 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:28:38.633815 (+ 2084us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:28:38.634361 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:28:38.634364 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:28:38.634364 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:28:38.634448 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:28:38.634707 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:28:38.634708 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:28:38.634708 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:28:38.634921 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:28:38.635003 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 09:28:38.635216 (+ 213us) server_negotiation.cc:299] Negotiation successful
0620 09:28:38.635331 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":706,"thread_start_us":71,"threads_started":1}
I20260620 09:28:38.635612 29945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:38.625501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:40067 (local address 127.0.0.1:54318)
0620 09:28:38.625698 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0620 09:28:38.625732 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 09:28:38.626161 (+ 429us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:28:38.626516 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:28:38.626520 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:28:38.626776 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:28:38.626937 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:28:38.626941 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:28:38.628203 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:28:38.628204 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:28:38.628877 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:28:38.628889 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:28:38.629401 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:28:38.629402 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:28:38.629476 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:28:38.630033 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:28:38.630041 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:28:38.631394 (+ 1353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:28:38.633937 (+ 2543us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:28:38.633944 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:28:38.633946 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:28:38.634270 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:28:38.634523 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:28:38.634524 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:28:38.634524 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:28:38.634641 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:28:38.634997 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:28:38.635005 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:28:38.635083 (+ 78us) client_negotiation.cc:769] Sending connection context
0620 09:28:38.635191 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 09:28:38.635414 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"mutex_wait_us":67,"thread_start_us":105,"threads_started":1}
I20260620 09:28:38.636545 29942 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.43.189" port: 45261 } has no permanent_uuid. Determining permanent_uuid...
I20260620 09:28:38.636938 29945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:38.636790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.189:45261 (local address 127.0.0.1:52656)
0620 09:28:38.636849 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0620 09:28:38.636911 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.189:45261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260620 09:28:38.637092 29893 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.43.189:45261: connect: Connection refused (error 111)
I20260620 09:28:38.637620 29945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:38.637508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.189:45261 (local address 127.0.0.1:52672)
0620 09:28:38.637556 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0620 09:28:38.637588 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.189:45261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260620 09:28:38.637737 29942 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.43.189:45261: Network error: Client connection negotiation failed: client connection to 127.16.43.189:45261: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.16.43.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.43.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.43.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.189.keytab.
Entry for principal kudu/127.16.43.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.189.keytab.
I20260620 09:28:38.659507 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781947562949448-16558-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781947562949448-16558-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781947562949448-16558-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781947562949448-16558-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.43.189:45261
--webserver_interface=127.16.43.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.189.keytab
--principal=kudu/127.16.43.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.43.148:41667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.43.190:40067,127.16.43.189:45261,127.16.43.188:41383
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781947562949448-16558-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781947562949448-16558-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781947562949448-16558-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 09:28:38.687014 29942 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.43.189" port: 45261 } attempt: 1
I20260620 09:28:38.687660 29945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:38.687492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.189:45261 (local address 127.0.0.1:52678)
0620 09:28:38.687574 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0620 09:28:38.687625 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.189:45261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260620 09:28:38.688035 29945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:38.687943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.189:45261 (local address 127.0.0.1:52694)
0620 09:28:38.687989 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0620 09:28:38.688016 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.189:45261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260620 09:28:38.688127 29942 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.43.189:45261: Network error: Client connection negotiation failed: client connection to 127.16.43.189:45261: connect: Connection refused (error 111)
W20260620 09:28:38.742242 29949 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:28:38.742465 29949 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:28:38.742487 29949 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:28:38.743594 29949 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:28:38.743630 29949 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:28:38.743644 29949 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:28:38.743657 29949 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:28:38.743669 29949 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:28:38.745118 29949 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:41667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781947562949448-16558-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781947562949448-16558-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.16.43.190:40067,127.16.43.189:45261,127.16.43.188:41383
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.189.keytab
--principal=kudu/127.16.43.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.43.189:45261
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781947562949448-16558-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.16.43.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29949
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781947562949448-16558-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:28:38.745311 29949 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:28:38.745496 29949 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:28:38.750469 29955 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:28:38.750521 29954 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:28:38.750504 29949 server_base.cc:1034] running on GCE node
W20260620 09:28:38.750478 29957 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:28:38.750881 29949 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:28:38.751166 29949 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:28:38.752300 29949 hybrid_clock.cc:648] HybridClock initialized: now 1781947718752286 us; error 38 us; skew 500 ppm
Jun 20 09:28:38 dist-test-slave-706q krb5kdc[29861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947718, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:28:38.754869 29949 init.cc:377] Logged in from keytab as kudu/127.16.43.189@KRBTEST.COM (short username kudu)
I20260620 09:28:38.755347 29949 webserver.cc:460] Webserver started at http://127.16.43.189:35531/ using document root <none> and password file <none>
I20260620 09:28:38.755647 29949 fs_manager.cc:359] Metadata directory not provided
I20260620 09:28:38.755717 29949 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:28:38.755851 29949 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:28:38.756263 29942 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.43.189" port: 45261 } attempt: 2
I20260620 09:28:38.756682 29945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:38.756551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.189:45261 (local address 127.0.0.1:52696)
0620 09:28:38.756613 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0620 09:28:38.756652 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.189:45261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260620 09:28:38.757068 29945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:38.756963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.189:45261 (local address 127.0.0.1:52708)
0620 09:28:38.757009 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0620 09:28:38.757040 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.189:45261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260620 09:28:38.757171 29942 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.43.189:45261: Network error: Client connection negotiation failed: client connection to 127.16.43.189:45261: connect: Connection refused (error 111)
I20260620 09:28:38.757205 29949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781947562949448-16558-0/minicluster-data/master-1/data/instance:
uuid: "6fedb9247f6a49c8b7f6a15745af0f60"
format_stamp: "Formatted at 2026-06-20 09:28:38 on dist-test-slave-706q"
server_key: "648a73f058f5eca2a2c78a4d07971065"
server_key_iv: "ac30d4363c2c74e7d3d283cdebc34d9e"
server_key_version: "encryptionkey@0"
I20260620 09:28:38.757604 29949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781947562949448-16558-0/minicluster-data/master-1/wal/instance:
uuid: "6fedb9247f6a49c8b7f6a15745af0f60"
format_stamp: "Formatted at 2026-06-20 09:28:38 on dist-test-slave-706q"
server_key: "648a73f058f5eca2a2c78a4d07971065"
server_key_iv: "ac30d4363c2c74e7d3d283cdebc34d9e"
server_key_version: "encryptionkey@0"
I20260620 09:28:38.758989 29949 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 09:28:38.759914 29964 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:28:38.760136 29949 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 09:28:38.760236 29949 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781947562949448-16558-0/minicluster-data/master-1/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781947562949448-16558-0/minicluster-data/master-1/wal
uuid: "6fedb9247f6a49c8b7f6a15745af0f60"
format_stamp: "Formatted at 2026-06-20 09:28:38 on dist-test-slave-706q"
server_key: "648a73f058f5eca2a2c78a4d07971065"
server_key_iv: "ac30d4363c2c74e7d3d283cdebc34d9e"
server_key_version: "encryptionkey@0"
I20260620 09:28:38.760294 29949 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781947562949448-16558-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781947562949448-16558-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781947562949448-16558-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:28:38.772405 29949 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:28:38.774638 29949 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:28:38.774765 29949 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:28:38.778853 29949 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.189:45261
I20260620 09:28:38.778915 30016 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.189:45261 every 8 connection(s)
I20260620 09:28:38.779273 29949 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781947562949448-16558-0/minicluster-data/master-1/data/info.pb
I20260620 09:28:38.780198 30017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:28:38.781872 30017 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.43.190" port: 40067 } has no permanent_uuid. Determining permanent_uuid...
I20260620 09:28:38.784518 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 29949
I20260620 09:28:38.784602 16558 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781947562949448-16558-0/minicluster-data/master-1/wal/instance
I20260620 09:28:38.784894 16558 external_mini_cluster.cc:1428] Setting key 4ea059da72dfc68888eda0672dbd3a4f
Jun 20 09:28:38 dist-test-slave-706q krb5kdc[29861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947718, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.189@KRBTEST.COM for kudu/127.16.43.190@KRBTEST.COM
I20260620 09:28:38.792989 29944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:38.782815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54332 (local address 127.16.43.190:40067)
0620 09:28:38.782908 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0620 09:28:38.782910 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:28:38.782977 (+ 67us) server_negotiation.cc:407] Connection header received
0620 09:28:38.783498 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:28:38.783500 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:28:38.783515 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:28:38.783630 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:28:38.784233 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:28:38.784864 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:28:38.786362 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:28:38.786798 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:28:38.786885 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:28:38.789511 (+ 2626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:28:38.789513 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:28:38.789513 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:28:38.789523 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:28:38.791368 (+ 1845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:28:38.791869 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:28:38.791870 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:28:38.791871 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:28:38.791943 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:28:38.792240 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:28:38.792241 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:28:38.792241 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:28:38.792409 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:28:38.792501 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 09:28:38.792758 (+ 257us) server_negotiation.cc:299] Negotiation successful
0620 09:28:38.792862 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 09:28:38.793031 30019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:38.782662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:40067 (local address 127.0.0.1:54332)
0620 09:28:38.782862 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0620 09:28:38.782913 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 09:28:38.783350 (+ 437us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:28:38.783705 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:28:38.783709 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:28:38.783956 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:28:38.784157 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:28:38.784161 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:28:38.784954 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:28:38.784955 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:28:38.786262 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:28:38.786266 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:28:38.786901 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:28:38.786902 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:28:38.787012 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:28:38.787670 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:28:38.787678 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:28:38.789346 (+ 1668us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:28:38.791469 (+ 2123us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:28:38.791476 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:28:38.791477 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:28:38.791787 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:28:38.792046 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:28:38.792047 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:28:38.792047 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:28:38.792162 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:28:38.792488 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:28:38.792493 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:28:38.792618 (+ 125us) client_negotiation.cc:769] Sending connection context
0620 09:28:38.792704 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 09:28:38.792837 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":76,"threads_started":1}
WARNING: no policy specified for kudu/127.16.43.188@KRBTEST.COM; defaulting to no policy
I20260620 09:28:38.793802 30017 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.43.189" port: 45261 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.43.188@KRBTEST.COM" created.
Jun 20 09:28:38 dist-test-slave-706q krb5kdc[29861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947718, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.189@KRBTEST.COM for kudu/127.16.43.189@KRBTEST.COM
I20260620 09:28:38.801802 30021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:38.794146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52722 (local address 127.16.43.189:45261)
0620 09:28:38.794319 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 09:28:38.794324 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:28:38.794342 (+ 18us) server_negotiation.cc:407] Connection header received
0620 09:28:38.794363 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:28:38.794364 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:28:38.794376 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:28:38.794445 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:28:38.794722 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:28:38.795520 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:28:38.796307 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:28:38.796651 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:28:38.796715 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:28:38.798764 (+ 2049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:28:38.798781 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:28:38.798781 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:28:38.798793 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:28:38.800477 (+ 1684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:28:38.800908 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:28:38.800911 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:28:38.800911 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:28:38.800977 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:28:38.801200 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:28:38.801201 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:28:38.801201 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:28:38.801369 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:28:38.801432 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 09:28:38.801621 (+ 189us) server_negotiation.cc:299] Negotiation successful
0620 09:28:38.801722 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":143,"thread_start_us":95,"threads_started":1}
I20260620 09:28:38.802031 30019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:38.794088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.189:45261 (local address 127.0.0.1:52722)
0620 09:28:38.794158 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0620 09:28:38.794168 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 09:28:38.794227 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:28:38.794459 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:28:38.794460 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:28:38.794578 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:28:38.794635 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:28:38.794636 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:28:38.795655 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:28:38.795656 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:28:38.796215 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:28:38.796217 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:28:38.796729 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:28:38.796730 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:28:38.796801 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:28:38.797258 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:28:38.797263 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:28:38.798672 (+ 1409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:28:38.800562 (+ 1890us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:28:38.800564 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:28:38.800564 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:28:38.800820 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:28:38.801049 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:28:38.801050 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:28:38.801051 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:28:38.801130 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:28:38.801437 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:28:38.801438 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:28:38.801505 (+ 67us) client_negotiation.cc:769] Sending connection context
0620 09:28:38.801599 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 09:28:38.801817 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54,"mutex_wait_us":88}
I20260620 09:28:38.802764 30017 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.43.188" port: 41383 } has no permanent_uuid. Determining permanent_uuid...
I20260620 09:28:38.803217 30019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:38.803070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.188:41383 (local address 127.0.0.1:43296)
0620 09:28:38.803124 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0620 09:28:38.803179 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.188:41383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260620 09:28:38.803367 29968 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.43.188:41383: connect: Connection refused (error 111)
I20260620 09:28:38.803841 30019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:38.803725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.188:41383 (local address 127.0.0.1:43306)
0620 09:28:38.803773 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0620 09:28:38.803807 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.188:41383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260620 09:28:38.803982 30017 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.43.188:41383: Network error: Client connection negotiation failed: client connection to 127.16.43.188:41383: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.43.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.188.keytab.
Entry for principal kudu/127.16.43.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.188.keytab.
I20260620 09:28:38.814510 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781947562949448-16558-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781947562949448-16558-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781947562949448-16558-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781947562949448-16558-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.43.188:41383
--webserver_interface=127.16.43.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.188.keytab
--principal=kudu/127.16.43.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.43.148:41667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.43.190:40067,127.16.43.189:45261,127.16.43.188:41383
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781947562949448-16558-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781947562949448-16558-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781947562949448-16558-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 09:28:38.848299 29942 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.43.189" port: 45261 } attempt: 3
Jun 20 09:28:38 dist-test-slave-706q krb5kdc[29861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947718, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.190@KRBTEST.COM for kudu/127.16.43.189@KRBTEST.COM
I20260620 09:28:38.853226 30017 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.43.188" port: 41383 } attempt: 1
I20260620 09:28:38.853715 30019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:38.853593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.188:41383 (local address 127.0.0.1:43320)
0620 09:28:38.853649 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0620 09:28:38.853679 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.188:41383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260620 09:28:38.854089 30019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:38.854000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.188:41383 (local address 127.0.0.1:43336)
0620 09:28:38.854041 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0620 09:28:38.854066 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.188:41383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260620 09:28:38.854254 30017 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.43.188:41383: Network error: Client connection negotiation failed: client connection to 127.16.43.188:41383: connect: Connection refused (error 111)
I20260620 09:28:38.856410 29945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:38.848707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.189:45261 (local address 127.0.0.1:52734)
0620 09:28:38.848791 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0620 09:28:38.848807 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 09:28:38.848891 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:28:38.849024 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:28:38.849025 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:28:38.849177 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:28:38.849244 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:28:38.849245 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:28:38.850244 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:28:38.850245 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:28:38.850791 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:28:38.850792 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:28:38.851275 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:28:38.851276 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:28:38.851330 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:28:38.851834 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:28:38.851839 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:28:38.853286 (+ 1447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:28:38.855421 (+ 2135us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:28:38.855422 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:28:38.855423 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:28:38.855686 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:28:38.855883 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:28:38.855883 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:28:38.855884 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:28:38.855933 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:28:38.856188 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:28:38.856189 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:28:38.856228 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 09:28:38.856275 (+ 47us) client_negotiation.cc:240] Negotiation successful
0620 09:28:38.856346 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260620 09:28:38.856458 30021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:38.848811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52734 (local address 127.16.43.189:45261)
0620 09:28:38.848887 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0620 09:28:38.848889 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:28:38.848902 (+ 13us) server_negotiation.cc:407] Connection header received
0620 09:28:38.848938 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:28:38.848939 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:28:38.848950 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:28:38.849012 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:28:38.849314 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:28:38.850153 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:28:38.850871 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:28:38.851199 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:28:38.851266 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:28:38.853376 (+ 2110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:28:38.853377 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:28:38.853378 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:28:38.853387 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:28:38.855319 (+ 1932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:28:38.855766 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:28:38.855767 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:28:38.855767 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:28:38.855827 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:28:38.855996 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:28:38.855997 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:28:38.855997 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:28:38.856128 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:28:38.856198 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 09:28:38.856299 (+ 101us) server_negotiation.cc:299] Negotiation successful
0620 09:28:38.856371 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 09:28:38.857000 29942 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.43.188" port: 41383 } has no permanent_uuid. Determining permanent_uuid...
I20260620 09:28:38.857326 29945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:38.857227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.188:41383 (local address 127.0.0.1:43346)
0620 09:28:38.857272 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0620 09:28:38.857297 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.188:41383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260620 09:28:38.857672 29945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:38.857612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.188:41383 (local address 127.0.0.1:43358)
0620 09:28:38.857638 (+ 26us) negotiation.cc:107] Waiting for socket to connect
0620 09:28:38.857658 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.188:41383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
W20260620 09:28:38.857762 29942 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.43.188:41383: Network error: Client connection negotiation failed: client connection to 127.16.43.188:41383: connect: Connection refused (error 111)
I20260620 09:28:38.888918 29942 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.43.188" port: 41383 } attempt: 1
I20260620 09:28:38.889503 29945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:38.889319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.188:41383 (local address 127.0.0.1:43372)
0620 09:28:38.889427 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0620 09:28:38.889471 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.188:41383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260620 09:28:38.889847 29945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:38.889750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.188:41383 (local address 127.0.0.1:43378)
0620 09:28:38.889792 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0620 09:28:38.889824 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.188:41383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260620 09:28:38.889940 29942 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.43.188:41383: Network error: Client connection negotiation failed: client connection to 127.16.43.188:41383: connect: Connection refused (error 111)
W20260620 09:28:38.900477 30024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:28:38.900729 30024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:28:38.900766 30024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:28:38.901979 30024 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:28:38.902047 30024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:28:38.902074 30024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:28:38.902104 30024 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:28:38.902133 30024 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:28:38.903707 30024 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:41667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781947562949448-16558-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781947562949448-16558-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.16.43.190:40067,127.16.43.189:45261,127.16.43.188:41383
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.188.keytab
--principal=kudu/127.16.43.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.43.188:41383
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781947562949448-16558-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.16.43.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781947562949448-16558-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:28:38.903995 30024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:28:38.904224 30024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:28:38.909413 30029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:28:38.909399 30030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:28:38.909399 30032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:28:38.909698 30024 server_base.cc:1034] running on GCE node
I20260620 09:28:38.909945 30024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:28:38.910269 30024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:28:38.911468 30024 hybrid_clock.cc:648] HybridClock initialized: now 1781947718911448 us; error 37 us; skew 500 ppm
Jun 20 09:28:38 dist-test-slave-706q krb5kdc[29861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947718, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:28:38.914026 30024 init.cc:377] Logged in from keytab as kudu/127.16.43.188@KRBTEST.COM (short username kudu)
I20260620 09:28:38.914546 30024 webserver.cc:460] Webserver started at http://127.16.43.188:37427/ using document root <none> and password file <none>
I20260620 09:28:38.914785 30024 fs_manager.cc:359] Metadata directory not provided
I20260620 09:28:38.914880 30024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:28:38.915040 30024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:28:38.916297 30024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781947562949448-16558-0/minicluster-data/master-2/data/instance:
uuid: "ad96ce9a54a24a5e9a20d318025b3bb6"
format_stamp: "Formatted at 2026-06-20 09:28:38 on dist-test-slave-706q"
server_key: "3cb818cfca3a84ec9a48fde908043bcf"
server_key_iv: "b3f71708e0f3c6e71bbde3e3c3c44b2a"
server_key_version: "encryptionkey@0"
I20260620 09:28:38.916707 30024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781947562949448-16558-0/minicluster-data/master-2/wal/instance:
uuid: "ad96ce9a54a24a5e9a20d318025b3bb6"
format_stamp: "Formatted at 2026-06-20 09:28:38 on dist-test-slave-706q"
server_key: "3cb818cfca3a84ec9a48fde908043bcf"
server_key_iv: "b3f71708e0f3c6e71bbde3e3c3c44b2a"
server_key_version: "encryptionkey@0"
I20260620 09:28:38.918170 30024 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 09:28:38.919070 30039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:28:38.919286 30024 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:28:38.919404 30024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781947562949448-16558-0/minicluster-data/master-2/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781947562949448-16558-0/minicluster-data/master-2/wal
uuid: "ad96ce9a54a24a5e9a20d318025b3bb6"
format_stamp: "Formatted at 2026-06-20 09:28:38 on dist-test-slave-706q"
server_key: "3cb818cfca3a84ec9a48fde908043bcf"
server_key_iv: "b3f71708e0f3c6e71bbde3e3c3c44b2a"
server_key_version: "encryptionkey@0"
I20260620 09:28:38.919481 30024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781947562949448-16558-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781947562949448-16558-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781947562949448-16558-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:28:38.922410 30017 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.43.188" port: 41383 } attempt: 2
I20260620 09:28:38.922879 30019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:38.922741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.188:41383 (local address 127.0.0.1:43382)
0620 09:28:38.922800 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0620 09:28:38.922844 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.188:41383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260620 09:28:38.923274 30019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:38.923186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.188:41383 (local address 127.0.0.1:43398)
0620 09:28:38.923222 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0620 09:28:38.923252 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.188:41383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260620 09:28:38.923398 30017 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.43.188:41383: Network error: Client connection negotiation failed: client connection to 127.16.43.188:41383: connect: Connection refused (error 111)
I20260620 09:28:38.933993 30024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:28:38.936447 30024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:28:38.936604 30024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:28:38.940634 30024 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.188:41383
I20260620 09:28:38.940703 30091 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.188:41383 every 8 connection(s)
I20260620 09:28:38.941046 30024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781947562949448-16558-0/minicluster-data/master-2/data/info.pb
I20260620 09:28:38.941965 30092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:28:38.943645 30092 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.43.190" port: 40067 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 09:28:38 dist-test-slave-706q krb5kdc[29861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947718, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.188@KRBTEST.COM for kudu/127.16.43.190@KRBTEST.COM
I20260620 09:28:38.949918 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 30024
I20260620 09:28:38.950029 16558 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781947562949448-16558-0/minicluster-data/master-2/wal/instance
I20260620 09:28:38.950275 16558 external_mini_cluster.cc:1428] Setting key 169232e5e010aec6b062d7c3222e11e5
I20260620 09:28:38.953537 29944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:38.944392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54340 (local address 127.16.43.190:40067)
0620 09:28:38.944533 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 09:28:38.944535 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:28:38.944689 (+ 154us) server_negotiation.cc:407] Connection header received
0620 09:28:38.945168 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:28:38.945170 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:28:38.945203 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:28:38.945294 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:28:38.945942 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:28:38.946595 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:28:38.947659 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:28:38.947863 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:28:38.947928 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:28:38.950252 (+ 2324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:28:38.950253 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:28:38.950254 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:28:38.950262 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:28:38.952079 (+ 1817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:28:38.952483 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:28:38.952491 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:28:38.952491 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:28:38.952556 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:28:38.952845 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:28:38.952846 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:28:38.952846 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:28:38.952991 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:28:38.953052 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 09:28:38.953389 (+ 337us) server_negotiation.cc:299] Negotiation successful
0620 09:28:38.953466 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260620 09:28:38.953778 30094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:38.944365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:40067 (local address 127.0.0.1:54340)
0620 09:28:38.944566 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0620 09:28:38.944614 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 09:28:38.945055 (+ 441us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:28:38.945357 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:28:38.945361 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:28:38.945650 (+ 289us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:28:38.945849 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:28:38.945854 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:28:38.946706 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:28:38.946707 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:28:38.947521 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:28:38.947575 (+ 54us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:28:38.947939 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:28:38.947940 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:28:38.948049 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:28:38.948609 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:28:38.948618 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:28:38.950104 (+ 1486us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:28:38.952156 (+ 2052us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:28:38.952164 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:28:38.952166 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:28:38.952395 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:28:38.952621 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:28:38.952622 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:28:38.952622 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:28:38.952740 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:28:38.953060 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:28:38.953068 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:28:38.953227 (+ 159us) client_negotiation.cc:769] Sending connection context
0620 09:28:38.953342 (+ 115us) client_negotiation.cc:240] Negotiation successful
0620 09:28:38.953519 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":90,"threads_started":1}
I20260620 09:28:38.954361 30092 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.43.189" port: 45261 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 09:28:38 dist-test-slave-706q krb5kdc[29861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947718, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.43.188@KRBTEST.COM
Jun 20 09:28:38 dist-test-slave-706q krb5kdc[29861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947718, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.188@KRBTEST.COM for kudu/127.16.43.189@KRBTEST.COM
I20260620 09:28:38.959641 30096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:38.950661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43408 (local address 127.16.43.188:41383)
0620 09:28:38.950854 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 09:28:38.950859 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:28:38.950917 (+ 58us) server_negotiation.cc:407] Connection header received
0620 09:28:38.950988 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:28:38.950990 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:28:38.951002 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:28:38.951071 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:28:38.951838 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:28:38.952692 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:28:38.953683 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:28:38.954109 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:28:38.954186 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:28:38.956774 (+ 2588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:28:38.956782 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:28:38.956782 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:28:38.956788 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:28:38.958379 (+ 1591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:28:38.958759 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:28:38.958762 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:28:38.958763 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:28:38.958818 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:28:38.959006 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:28:38.959007 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:28:38.959008 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:28:38.959172 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:28:38.959312 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 09:28:38.959415 (+ 103us) server_negotiation.cc:299] Negotiation successful
0620 09:28:38.959498 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":98,"threads_started":1}
I20260620 09:28:38.961396 30021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:38.954758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52748 (local address 127.16.43.189:45261)
0620 09:28:38.954819 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 09:28:38.954821 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:28:38.954830 (+ 9us) server_negotiation.cc:407] Connection header received
0620 09:28:38.954853 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:28:38.954854 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:28:38.954864 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:28:38.954916 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:28:38.955224 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:28:38.955810 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:28:38.956521 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:28:38.956841 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:28:38.956897 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:28:38.958689 (+ 1792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:28:38.958690 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:28:38.958690 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:28:38.958696 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:28:38.960250 (+ 1554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:28:38.960651 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:28:38.960653 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:28:38.960653 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:28:38.960712 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:28:38.960900 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:28:38.960901 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:28:38.960901 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:28:38.961024 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:28:38.961098 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 09:28:38.961237 (+ 139us) server_negotiation.cc:299] Negotiation successful
0620 09:28:38.961308 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 09:28:38.961396 30094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:38.954646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.189:45261 (local address 127.0.0.1:52748)
0620 09:28:38.954696 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0620 09:28:38.954707 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 09:28:38.954746 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:28:38.954942 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:28:38.954943 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:28:38.955092 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:28:38.955151 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:28:38.955153 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:28:38.955890 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:28:38.955891 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:28:38.956436 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:28:38.956438 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:28:38.956910 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:28:38.956911 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:28:38.957000 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:28:38.957318 (+ 318us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:28:38.957323 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:28:38.958610 (+ 1287us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:28:38.960322 (+ 1712us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:28:38.960322 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:28:38.960323 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:28:38.960580 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:28:38.960775 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:28:38.960776 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:28:38.960776 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:28:38.960832 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:28:38.961089 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:28:38.961090 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:28:38.961131 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 09:28:38.961212 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 09:28:38.961303 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260620 09:28:38.961704 30092 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.43.188" port: 41383 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 09:28:38 dist-test-slave-706q krb5kdc[29861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947718, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.43.190@KRBTEST.COM
I20260620 09:28:38.965116 29942 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.43.188" port: 41383 } attempt: 2
Jun 20 09:28:38 dist-test-slave-706q krb5kdc[29861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947718, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.188@KRBTEST.COM for kudu/127.16.43.188@KRBTEST.COM
I20260620 09:28:38.968571 29944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:38.960546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54352 (local address 127.16.43.190:40067)
0620 09:28:38.960592 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 09:28:38.960594 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:28:38.960613 (+ 19us) server_negotiation.cc:407] Connection header received
0620 09:28:38.960633 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:28:38.960634 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:28:38.960643 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:28:38.960710 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:28:38.961206 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:28:38.961875 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:28:38.962735 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:28:38.962938 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:28:38.963010 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:28:38.965118 (+ 2108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:28:38.965119 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:28:38.965120 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:28:38.965128 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:28:38.967063 (+ 1935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:28:38.967563 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:28:38.967565 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:28:38.967565 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:28:38.967624 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:28:38.967848 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:28:38.967849 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:28:38.967849 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:28:38.967991 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:28:38.968189 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0620 09:28:38.968382 (+ 193us) server_negotiation.cc:299] Negotiation successful
0620 09:28:38.968456 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 09:28:38.968942 30094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:38.961968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.188:41383 (local address 127.0.0.1:43422)
0620 09:28:38.962053 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0620 09:28:38.962062 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 09:28:38.962104 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:28:38.962206 (+ 102us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:28:38.962206 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:28:38.962345 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:28:38.962376 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:28:38.962378 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:28:38.963103 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:28:38.963105 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:28:38.963799 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:28:38.963801 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:28:38.964188 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:28:38.964190 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:28:38.964237 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:28:38.964602 (+ 365us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:28:38.964607 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:28:38.965874 (+ 1267us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:28:38.967908 (+ 2034us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:28:38.967909 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:28:38.967909 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:28:38.968101 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:28:38.968295 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:28:38.968296 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:28:38.968296 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:28:38.968372 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:28:38.968630 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:28:38.968632 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:28:38.968688 (+ 56us) client_negotiation.cc:769] Sending connection context
0620 09:28:38.968762 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 09:28:38.968841 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260620 09:28:38.969084 30096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:38.962014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43422 (local address 127.16.43.188:41383)
0620 09:28:38.962048 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0620 09:28:38.962054 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 09:28:38.962091 (+ 37us) server_negotiation.cc:407] Connection header received
0620 09:28:38.962155 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:28:38.962155 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:28:38.962163 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:28:38.962202 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:28:38.962498 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:28:38.963006 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:28:38.963902 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:28:38.964098 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:28:38.964170 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:28:38.966219 (+ 2049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:28:38.966221 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:28:38.966221 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:28:38.966228 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:28:38.967830 (+ 1602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:28:38.968174 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:28:38.968174 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:28:38.968175 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:28:38.968222 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:28:38.968452 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:28:38.968453 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:28:38.968453 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:28:38.968571 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:28:38.968633 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 09:28:38.968808 (+ 175us) server_negotiation.cc:299] Negotiation successful
0620 09:28:38.968934 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":18}
I20260620 09:28:38.970645 30092 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad96ce9a54a24a5e9a20d318025b3bb6: Bootstrap starting.
Jun 20 09:28:38 dist-test-slave-706q krb5kdc[29861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947718, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.190@KRBTEST.COM for kudu/127.16.43.188@KRBTEST.COM
I20260620 09:28:38.971395 30092 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ad96ce9a54a24a5e9a20d318025b3bb6: Neither blocks nor log segments found. Creating new log.
I20260620 09:28:38.971802 30092 log.cc:824] T 00000000000000000000000000000000 P ad96ce9a54a24a5e9a20d318025b3bb6: Log is configured to *not* fsync() on all Append() calls
I20260620 09:28:38.972728 30092 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad96ce9a54a24a5e9a20d318025b3bb6: No bootstrap required, opened a new log
I20260620 09:28:38.974522 29945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:38.965766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.188:41383 (local address 127.0.0.1:43428)
0620 09:28:38.965815 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0620 09:28:38.965825 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 09:28:38.965871 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:28:38.966220 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:28:38.966221 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:28:38.966502 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:28:38.966554 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:28:38.966556 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:28:38.967991 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:28:38.967992 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:28:38.968783 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:28:38.968785 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:28:38.969585 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:28:38.969587 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:28:38.969662 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:28:38.970034 (+ 372us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:28:38.970041 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:28:38.971429 (+ 1388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:28:38.973312 (+ 1883us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:28:38.973314 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:28:38.973314 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:28:38.973563 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:28:38.973803 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:28:38.973804 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:28:38.973805 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:28:38.973869 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:28:38.974157 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:28:38.974158 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:28:38.974208 (+ 50us) client_negotiation.cc:769] Sending connection context
0620 09:28:38.974297 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 09:28:38.974407 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260620 09:28:38.974558 30097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:38.965790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43428 (local address 127.16.43.188:41383)
0620 09:28:38.966065 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0620 09:28:38.966067 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:28:38.966076 (+ 9us) server_negotiation.cc:407] Connection header received
0620 09:28:38.966094 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:28:38.966094 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:28:38.966103 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:28:38.966214 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:28:38.966677 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:28:38.967904 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:28:38.968873 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:28:38.969473 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:28:38.969814 (+ 341us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:28:38.971582 (+ 1768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:28:38.971583 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:28:38.971583 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:28:38.971591 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:28:38.973226 (+ 1635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:28:38.973645 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:28:38.973647 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:28:38.973647 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:28:38.973730 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:28:38.973944 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:28:38.973945 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:28:38.973945 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:28:38.974084 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:28:38.974164 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 09:28:38.974325 (+ 161us) server_negotiation.cc:299] Negotiation successful
0620 09:28:38.974409 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":113,"threads_started":1}
Jun 20 09:28:38 dist-test-slave-706q krb5kdc[29861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947718, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.43.189@KRBTEST.COM
I20260620 09:28:38.975044 30092 raft_consensus.cc:348] T 00000000000000000000000000000000 P ad96ce9a54a24a5e9a20d318025b3bb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1cebf2b7d7b40dc85e05405ed6a8369" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 40067 } } peers { permanent_uuid: "6fedb9247f6a49c8b7f6a15745af0f60" member_type: VOTER last_known_addr { host: "127.16.43.189" port: 45261 } } peers { permanent_uuid: "ad96ce9a54a24a5e9a20d318025b3bb6" member_type: VOTER last_known_addr { host: "127.16.43.188" port: 41383 } }
I20260620 09:28:38.975191 30092 raft_consensus.cc:374] T 00000000000000000000000000000000 P ad96ce9a54a24a5e9a20d318025b3bb6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:28:38.975278 30092 raft_consensus.cc:729] T 00000000000000000000000000000000 P ad96ce9a54a24a5e9a20d318025b3bb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad96ce9a54a24a5e9a20d318025b3bb6, State: Initialized, Role: FOLLOWER
I20260620 09:28:38.975447 30092 consensus_queue.cc:260] T 00000000000000000000000000000000 P ad96ce9a54a24a5e9a20d318025b3bb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1cebf2b7d7b40dc85e05405ed6a8369" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 40067 } } peers { permanent_uuid: "6fedb9247f6a49c8b7f6a15745af0f60" member_type: VOTER last_known_addr { host: "127.16.43.189" port: 45261 } } peers { permanent_uuid: "ad96ce9a54a24a5e9a20d318025b3bb6" member_type: VOTER last_known_addr { host: "127.16.43.188" port: 41383 } }
I20260620 09:28:38.975847 30092 sys_catalog.cc:564] T 00000000000000000000000000000000 P ad96ce9a54a24a5e9a20d318025b3bb6 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:28:38.976701 29942 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1cebf2b7d7b40dc85e05405ed6a8369: Bootstrap starting.
I20260620 09:28:38.977540 29942 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e1cebf2b7d7b40dc85e05405ed6a8369: Neither blocks nor log segments found. Creating new log.
I20260620 09:28:38.977897 29942 log.cc:824] T 00000000000000000000000000000000 P e1cebf2b7d7b40dc85e05405ed6a8369: Log is configured to *not* fsync() on all Append() calls
I20260620 09:28:38.977440 30099 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad96ce9a54a24a5e9a20d318025b3bb6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1cebf2b7d7b40dc85e05405ed6a8369" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 40067 } } peers { permanent_uuid: "6fedb9247f6a49c8b7f6a15745af0f60" member_type: VOTER last_known_addr { host: "127.16.43.189" port: 45261 } } peers { permanent_uuid: "ad96ce9a54a24a5e9a20d318025b3bb6" member_type: VOTER last_known_addr { host: "127.16.43.188" port: 41383 } } }
I20260620 09:28:38.978657 30099 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad96ce9a54a24a5e9a20d318025b3bb6 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 09:28:38.978933 30021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:38.969559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52750 (local address 127.16.43.189:45261)
0620 09:28:38.969599 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 09:28:38.969601 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:28:38.969629 (+ 28us) server_negotiation.cc:407] Connection header received
0620 09:28:38.969653 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:28:38.969654 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:28:38.969664 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:28:38.969714 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:28:38.970405 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:28:38.971073 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:28:38.972416 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:28:38.972930 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:28:38.972988 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:28:38.975694 (+ 2706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:28:38.975695 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:28:38.975695 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:28:38.975703 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:28:38.977545 (+ 1842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:28:38.977915 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:28:38.977916 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:28:38.977916 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:28:38.977974 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:28:38.978169 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:28:38.978170 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:28:38.978170 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:28:38.978297 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:28:38.978505 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0620 09:28:38.978755 (+ 250us) server_negotiation.cc:299] Negotiation successful
0620 09:28:38.978833 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 09:28:38.979054 29942 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1cebf2b7d7b40dc85e05405ed6a8369: No bootstrap required, opened a new log
W20260620 09:28:38.977926 30110 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ad96ce9a54a24a5e9a20d318025b3bb6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 09:28:38.980330 30110 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 09:28:38.980636 16558 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 09:28:38.978958 30024 master_runner.cc:186] Error getting master registration for 127.16.43.190:40067: OK, instance_id { permanent_uuid: "e1cebf2b7d7b40dc85e05405ed6a8369" instance_seqno: 1781947718620562 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 09:28:38.981568 30024 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 09:28:38.981600 29942 raft_consensus.cc:348] T 00000000000000000000000000000000 P e1cebf2b7d7b40dc85e05405ed6a8369 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1cebf2b7d7b40dc85e05405ed6a8369" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 40067 } } peers { permanent_uuid: "6fedb9247f6a49c8b7f6a15745af0f60" member_type: VOTER last_known_addr { host: "127.16.43.189" port: 45261 } } peers { permanent_uuid: "ad96ce9a54a24a5e9a20d318025b3bb6" member_type: VOTER last_known_addr { host: "127.16.43.188" port: 41383 } }
I20260620 09:28:38.981745 29942 raft_consensus.cc:374] T 00000000000000000000000000000000 P e1cebf2b7d7b40dc85e05405ed6a8369 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:28:38.981823 29942 raft_consensus.cc:729] T 00000000000000000000000000000000 P e1cebf2b7d7b40dc85e05405ed6a8369 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1cebf2b7d7b40dc85e05405ed6a8369, State: Initialized, Role: FOLLOWER
I20260620 09:28:38.981969 29942 consensus_queue.cc:260] T 00000000000000000000000000000000 P e1cebf2b7d7b40dc85e05405ed6a8369 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1cebf2b7d7b40dc85e05405ed6a8369" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 40067 } } peers { permanent_uuid: "6fedb9247f6a49c8b7f6a15745af0f60" member_type: VOTER last_known_addr { host: "127.16.43.189" port: 45261 } } peers { permanent_uuid: "ad96ce9a54a24a5e9a20d318025b3bb6" member_type: VOTER last_known_addr { host: "127.16.43.188" port: 41383 } }
I20260620 09:28:38.982281 29942 sys_catalog.cc:564] T 00000000000000000000000000000000 P e1cebf2b7d7b40dc85e05405ed6a8369 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:28:38.982260 30118 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1cebf2b7d7b40dc85e05405ed6a8369 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1cebf2b7d7b40dc85e05405ed6a8369" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 40067 } } peers { permanent_uuid: "6fedb9247f6a49c8b7f6a15745af0f60" member_type: VOTER last_known_addr { host: "127.16.43.189" port: 45261 } } peers { permanent_uuid: "ad96ce9a54a24a5e9a20d318025b3bb6" member_type: VOTER last_known_addr { host: "127.16.43.188" port: 41383 } } }
I20260620 09:28:38.982388 30118 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1cebf2b7d7b40dc85e05405ed6a8369 [sys.catalog]: This master's current role is: FOLLOWER
W20260620 09:28:38.984141 30129 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e1cebf2b7d7b40dc85e05405ed6a8369: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 09:28:38.984202 30129 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 09:28:38.984946 29874 master_runner.cc:186] Error getting master registration for 127.16.43.189:45261: OK, instance_id { permanent_uuid: "6fedb9247f6a49c8b7f6a15745af0f60" instance_seqno: 1781947718777751 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 09:28:38.988133 29874 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 09:28:38.996631 30021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:38.987790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52764 (local address 127.16.43.189:45261)
0620 09:28:38.987836 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 09:28:38.987838 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:28:38.988344 (+ 506us) server_negotiation.cc:407] Connection header received
0620 09:28:38.988418 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:28:38.988419 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:28:38.988431 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:28:38.988488 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:28:38.989168 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:28:38.989910 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:28:38.990980 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:28:38.991269 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:28:38.991311 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:28:38.993055 (+ 1744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:28:38.993056 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:28:38.993057 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:28:38.993064 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:28:38.994795 (+ 1731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:28:38.995183 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:28:38.995184 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:28:38.995185 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:28:38.995238 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:28:38.995420 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:28:38.995421 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:28:38.995421 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:28:38.995586 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:28:38.996178 (+ 592us) server_negotiation.cc:1035] Waiting for connection context
0620 09:28:38.996437 (+ 259us) server_negotiation.cc:299] Negotiation successful
0620 09:28:38.996531 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 09:28:38.996695 29944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:38.981916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54368 (local address 127.16.43.190:40067)
0620 09:28:38.981963 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 09:28:38.981965 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:28:38.988181 (+ 6216us) server_negotiation.cc:407] Connection header received
0620 09:28:38.988256 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:28:38.988257 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:28:38.988269 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:28:38.988331 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:28:38.989117 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:28:38.989950 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:28:38.991089 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:28:38.991410 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:28:38.991457 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:28:38.993201 (+ 1744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:28:38.993202 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:28:38.993202 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:28:38.993210 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:28:38.995214 (+ 2004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:28:38.995730 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:28:38.995732 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:28:38.995732 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:28:38.995792 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:28:38.996035 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:28:38.996036 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:28:38.996036 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:28:38.996170 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:28:38.996278 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 09:28:38.996511 (+ 233us) server_negotiation.cc:299] Negotiation successful
0620 09:28:38.996595 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 09:28:39.003309 30097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:38.988028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43432 (local address 127.16.43.188:41383)
0620 09:28:38.988070 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 09:28:38.988072 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:28:38.995737 (+ 7665us) server_negotiation.cc:407] Connection header received
0620 09:28:38.995844 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:28:38.995846 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:28:38.995861 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:28:38.995897 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:28:38.996562 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:28:38.997183 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:28:38.998077 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:28:38.998523 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:28:38.998608 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:28:39.000117 (+ 1509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:28:39.000119 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:28:39.000119 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:28:39.000127 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:28:39.001962 (+ 1835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:28:39.002392 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:28:39.002394 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:28:39.002394 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:28:39.002460 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:28:39.002688 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:28:39.002688 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:28:39.002689 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:28:39.002839 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:28:39.002974 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 09:28:39.003084 (+ 110us) server_negotiation.cc:299] Negotiation successful
0620 09:28:39.003175 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 09:28:39.014573 30017 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.43.188" port: 41383 } attempt: 3
Jun 20 09:28:39 dist-test-slave-706q krb5kdc[29861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947718, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.189@KRBTEST.COM for kudu/127.16.43.188@KRBTEST.COM
I20260620 09:28:39.023685 30097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:39.014964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43438 (local address 127.16.43.188:41383)
0620 09:28:39.015009 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 09:28:39.015012 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:28:39.015021 (+ 9us) server_negotiation.cc:407] Connection header received
0620 09:28:39.015070 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:28:39.015071 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:28:39.015081 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:28:39.015138 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:28:39.015484 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:28:39.016377 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:28:39.017384 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:28:39.017737 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:28:39.017819 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:28:39.020273 (+ 2454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:28:39.020275 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:28:39.020275 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:28:39.020283 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:28:39.022293 (+ 2010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:28:39.022701 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:28:39.022703 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:28:39.022703 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:28:39.022771 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:28:39.023010 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:28:39.023012 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:28:39.023012 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:28:39.023193 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:28:39.023260 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 09:28:39.023478 (+ 218us) server_negotiation.cc:299] Negotiation successful
0620 09:28:39.023582 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 09:28:39.023725 30019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:39.014902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.188:41383 (local address 127.0.0.1:43438)
0620 09:28:39.014957 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0620 09:28:39.014970 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 09:28:39.015012 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:28:39.015151 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:28:39.015151 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:28:39.015342 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:28:39.015399 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:28:39.015401 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:28:39.016480 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:28:39.016481 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:28:39.017293 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:28:39.017295 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:28:39.017824 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:28:39.017825 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:28:39.017887 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:28:39.018424 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:28:39.018431 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:28:39.020163 (+ 1732us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:28:39.022375 (+ 2212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:28:39.022377 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:28:39.022377 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:28:39.022617 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:28:39.022843 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:28:39.022844 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:28:39.022844 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:28:39.022909 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:28:39.023265 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:28:39.023267 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:28:39.023358 (+ 91us) client_negotiation.cc:769] Sending connection context
0620 09:28:39.023461 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 09:28:39.023605 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260620 09:28:39.025781 30017 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fedb9247f6a49c8b7f6a15745af0f60: Bootstrap starting.
I20260620 09:28:39.026567 30017 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6fedb9247f6a49c8b7f6a15745af0f60: Neither blocks nor log segments found. Creating new log.
I20260620 09:28:39.026868 30017 log.cc:824] T 00000000000000000000000000000000 P 6fedb9247f6a49c8b7f6a15745af0f60: Log is configured to *not* fsync() on all Append() calls
I20260620 09:28:39.027825 30017 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fedb9247f6a49c8b7f6a15745af0f60: No bootstrap required, opened a new log
I20260620 09:28:39.030131 30017 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6fedb9247f6a49c8b7f6a15745af0f60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1cebf2b7d7b40dc85e05405ed6a8369" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 40067 } } peers { permanent_uuid: "6fedb9247f6a49c8b7f6a15745af0f60" member_type: VOTER last_known_addr { host: "127.16.43.189" port: 45261 } } peers { permanent_uuid: "ad96ce9a54a24a5e9a20d318025b3bb6" member_type: VOTER last_known_addr { host: "127.16.43.188" port: 41383 } }
I20260620 09:28:39.030277 30017 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6fedb9247f6a49c8b7f6a15745af0f60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:28:39.030311 30017 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6fedb9247f6a49c8b7f6a15745af0f60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fedb9247f6a49c8b7f6a15745af0f60, State: Initialized, Role: FOLLOWER
I20260620 09:28:39.030453 30017 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6fedb9247f6a49c8b7f6a15745af0f60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1cebf2b7d7b40dc85e05405ed6a8369" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 40067 } } peers { permanent_uuid: "6fedb9247f6a49c8b7f6a15745af0f60" member_type: VOTER last_known_addr { host: "127.16.43.189" port: 45261 } } peers { permanent_uuid: "ad96ce9a54a24a5e9a20d318025b3bb6" member_type: VOTER last_known_addr { host: "127.16.43.188" port: 41383 } }
I20260620 09:28:39.030758 30134 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fedb9247f6a49c8b7f6a15745af0f60 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1cebf2b7d7b40dc85e05405ed6a8369" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 40067 } } peers { permanent_uuid: "6fedb9247f6a49c8b7f6a15745af0f60" member_type: VOTER last_known_addr { host: "127.16.43.189" port: 45261 } } peers { permanent_uuid: "ad96ce9a54a24a5e9a20d318025b3bb6" member_type: VOTER last_known_addr { host: "127.16.43.188" port: 41383 } } }
I20260620 09:28:39.030812 30017 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6fedb9247f6a49c8b7f6a15745af0f60 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:28:39.030849 30134 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fedb9247f6a49c8b7f6a15745af0f60 [sys.catalog]: This master's current role is: FOLLOWER
W20260620 09:28:39.033078 30145 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6fedb9247f6a49c8b7f6a15745af0f60: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 09:28:39.033208 30145 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 09:28:39.035913 29949 master_runner.cc:263] No leader master found from master 6fedb9247f6a49c8b7f6a15745af0f60
I20260620 09:28:39.035966 29949 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260620 09:28:39.118611 29895 tablet.cc:2369] T 00000000000000000000000000000000 P e1cebf2b7d7b40dc85e05405ed6a8369: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 09:28:39.187255 30045 tablet.cc:2369] T 00000000000000000000000000000000 P ad96ce9a54a24a5e9a20d318025b3bb6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 09:28:39.217227 30118 raft_consensus.cc:482] T 00000000000000000000000000000000 P e1cebf2b7d7b40dc85e05405ed6a8369 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:28:39.217350 30118 raft_consensus.cc:504] T 00000000000000000000000000000000 P e1cebf2b7d7b40dc85e05405ed6a8369 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1cebf2b7d7b40dc85e05405ed6a8369" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 40067 } } peers { permanent_uuid: "6fedb9247f6a49c8b7f6a15745af0f60" member_type: VOTER last_known_addr { host: "127.16.43.189" port: 45261 } } peers { permanent_uuid: "ad96ce9a54a24a5e9a20d318025b3bb6" member_type: VOTER last_known_addr { host: "127.16.43.188" port: 41383 } }
I20260620 09:28:39.217643 30118 leader_election.cc:288] T 00000000000000000000000000000000 P e1cebf2b7d7b40dc85e05405ed6a8369 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6fedb9247f6a49c8b7f6a15745af0f60 (127.16.43.189:45261), ad96ce9a54a24a5e9a20d318025b3bb6 (127.16.43.188:41383)
I20260620 09:28:39.217998 29991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e1cebf2b7d7b40dc85e05405ed6a8369" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fedb9247f6a49c8b7f6a15745af0f60" is_pre_election: true
I20260620 09:28:39.218012 30066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e1cebf2b7d7b40dc85e05405ed6a8369" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad96ce9a54a24a5e9a20d318025b3bb6" is_pre_election: true
I20260620 09:28:39.218155 29991 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6fedb9247f6a49c8b7f6a15745af0f60 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1cebf2b7d7b40dc85e05405ed6a8369 in term 0.
I20260620 09:28:39.218202 30066 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ad96ce9a54a24a5e9a20d318025b3bb6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1cebf2b7d7b40dc85e05405ed6a8369 in term 0.
I20260620 09:28:39.218349 29893 leader_election.cc:302] T 00000000000000000000000000000000 P e1cebf2b7d7b40dc85e05405ed6a8369 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6fedb9247f6a49c8b7f6a15745af0f60, e1cebf2b7d7b40dc85e05405ed6a8369; no voters:
I20260620 09:28:39.218545 30118 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e1cebf2b7d7b40dc85e05405ed6a8369 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 09:28:39.218633 30118 raft_consensus.cc:482] T 00000000000000000000000000000000 P e1cebf2b7d7b40dc85e05405ed6a8369 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 09:28:39.218672 30118 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e1cebf2b7d7b40dc85e05405ed6a8369 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:28:39.219419 30118 raft_consensus.cc:504] T 00000000000000000000000000000000 P e1cebf2b7d7b40dc85e05405ed6a8369 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1cebf2b7d7b40dc85e05405ed6a8369" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 40067 } } peers { permanent_uuid: "6fedb9247f6a49c8b7f6a15745af0f60" member_type: VOTER last_known_addr { host: "127.16.43.189" port: 45261 } } peers { permanent_uuid: "ad96ce9a54a24a5e9a20d318025b3bb6" member_type: VOTER last_known_addr { host: "127.16.43.188" port: 41383 } }
I20260620 09:28:39.219580 30118 leader_election.cc:288] T 00000000000000000000000000000000 P e1cebf2b7d7b40dc85e05405ed6a8369 [CANDIDATE]: Term 1 election: Requested vote from peers 6fedb9247f6a49c8b7f6a15745af0f60 (127.16.43.189:45261), ad96ce9a54a24a5e9a20d318025b3bb6 (127.16.43.188:41383)
I20260620 09:28:39.219717 29991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e1cebf2b7d7b40dc85e05405ed6a8369" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fedb9247f6a49c8b7f6a15745af0f60"
I20260620 09:28:39.219758 30066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e1cebf2b7d7b40dc85e05405ed6a8369" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad96ce9a54a24a5e9a20d318025b3bb6"
I20260620 09:28:39.219795 29991 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6fedb9247f6a49c8b7f6a15745af0f60 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:28:39.219869 30066 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ad96ce9a54a24a5e9a20d318025b3bb6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:28:39.220553 29991 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6fedb9247f6a49c8b7f6a15745af0f60 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1cebf2b7d7b40dc85e05405ed6a8369 in term 1.
I20260620 09:28:39.220615 30066 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ad96ce9a54a24a5e9a20d318025b3bb6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1cebf2b7d7b40dc85e05405ed6a8369 in term 1.
I20260620 09:28:39.220764 29893 leader_election.cc:302] T 00000000000000000000000000000000 P e1cebf2b7d7b40dc85e05405ed6a8369 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6fedb9247f6a49c8b7f6a15745af0f60, e1cebf2b7d7b40dc85e05405ed6a8369; no voters:
I20260620 09:28:39.220886 30118 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e1cebf2b7d7b40dc85e05405ed6a8369 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:28:39.221019 30118 raft_consensus.cc:686] T 00000000000000000000000000000000 P e1cebf2b7d7b40dc85e05405ed6a8369 [term 1 LEADER]: Becoming Leader. State: Replica: e1cebf2b7d7b40dc85e05405ed6a8369, State: Running, Role: LEADER
I20260620 09:28:39.221119 30118 consensus_queue.cc:237] T 00000000000000000000000000000000 P e1cebf2b7d7b40dc85e05405ed6a8369 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1cebf2b7d7b40dc85e05405ed6a8369" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 40067 } } peers { permanent_uuid: "6fedb9247f6a49c8b7f6a15745af0f60" member_type: VOTER last_known_addr { host: "127.16.43.189" port: 45261 } } peers { permanent_uuid: "ad96ce9a54a24a5e9a20d318025b3bb6" member_type: VOTER last_known_addr { host: "127.16.43.188" port: 41383 } }
I20260620 09:28:39.221618 30147 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1cebf2b7d7b40dc85e05405ed6a8369 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e1cebf2b7d7b40dc85e05405ed6a8369. Latest consensus state: current_term: 1 leader_uuid: "e1cebf2b7d7b40dc85e05405ed6a8369" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1cebf2b7d7b40dc85e05405ed6a8369" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 40067 } } peers { permanent_uuid: "6fedb9247f6a49c8b7f6a15745af0f60" member_type: VOTER last_known_addr { host: "127.16.43.189" port: 45261 } } peers { permanent_uuid: "ad96ce9a54a24a5e9a20d318025b3bb6" member_type: VOTER last_known_addr { host: "127.16.43.188" port: 41383 } } }
I20260620 09:28:39.221706 30147 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1cebf2b7d7b40dc85e05405ed6a8369 [sys.catalog]: This master's current role is: LEADER
I20260620 09:28:39.221935 30149 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:28:39.222124 30149 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:28:39.223598 29991 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6fedb9247f6a49c8b7f6a15745af0f60 [term 1 FOLLOWER]: Refusing update from remote peer e1cebf2b7d7b40dc85e05405ed6a8369: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:28:39.223610 30066 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ad96ce9a54a24a5e9a20d318025b3bb6 [term 1 FOLLOWER]: Refusing update from remote peer e1cebf2b7d7b40dc85e05405ed6a8369: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:28:39.223901 30118 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e1cebf2b7d7b40dc85e05405ed6a8369 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad96ce9a54a24a5e9a20d318025b3bb6" member_type: VOTER last_known_addr { host: "127.16.43.188" port: 41383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:28:39.224021 30147 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e1cebf2b7d7b40dc85e05405ed6a8369 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6fedb9247f6a49c8b7f6a15745af0f60" member_type: VOTER last_known_addr { host: "127.16.43.189" port: 45261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:28:39.225160 30099 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad96ce9a54a24a5e9a20d318025b3bb6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e1cebf2b7d7b40dc85e05405ed6a8369. Latest consensus state: current_term: 1 leader_uuid: "e1cebf2b7d7b40dc85e05405ed6a8369" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1cebf2b7d7b40dc85e05405ed6a8369" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 40067 } } peers { permanent_uuid: "6fedb9247f6a49c8b7f6a15745af0f60" member_type: VOTER last_known_addr { host: "127.16.43.189" port: 45261 } } peers { permanent_uuid: "ad96ce9a54a24a5e9a20d318025b3bb6" member_type: VOTER last_known_addr { host: "127.16.43.188" port: 41383 } } }
I20260620 09:28:39.225176 30134 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fedb9247f6a49c8b7f6a15745af0f60 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e1cebf2b7d7b40dc85e05405ed6a8369. Latest consensus state: current_term: 1 leader_uuid: "e1cebf2b7d7b40dc85e05405ed6a8369" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1cebf2b7d7b40dc85e05405ed6a8369" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 40067 } } peers { permanent_uuid: "6fedb9247f6a49c8b7f6a15745af0f60" member_type: VOTER last_known_addr { host: "127.16.43.189" port: 45261 } } peers { permanent_uuid: "ad96ce9a54a24a5e9a20d318025b3bb6" member_type: VOTER last_known_addr { host: "127.16.43.188" port: 41383 } } }
I20260620 09:28:39.225260 30099 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad96ce9a54a24a5e9a20d318025b3bb6 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 09:28:39.225265 30134 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fedb9247f6a49c8b7f6a15745af0f60 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 09:28:39.226346 30150 mvcc.cc:205] Tried to move back new op lower bound from 7298857857937178624 to 7298857857930317824. Current Snapshot: MvccSnapshot[applied={T|T < 7298857857937178624}]
I20260620 09:28:39.226220 30118 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1cebf2b7d7b40dc85e05405ed6a8369 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e1cebf2b7d7b40dc85e05405ed6a8369" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1cebf2b7d7b40dc85e05405ed6a8369" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 40067 } } peers { permanent_uuid: "6fedb9247f6a49c8b7f6a15745af0f60" member_type: VOTER last_known_addr { host: "127.16.43.189" port: 45261 } } peers { permanent_uuid: "ad96ce9a54a24a5e9a20d318025b3bb6" member_type: VOTER last_known_addr { host: "127.16.43.188" port: 41383 } } }
I20260620 09:28:39.226438 30118 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1cebf2b7d7b40dc85e05405ed6a8369 [sys.catalog]: This master's current role is: LEADER
I20260620 09:28:39.226490 30118 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1cebf2b7d7b40dc85e05405ed6a8369 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e1cebf2b7d7b40dc85e05405ed6a8369" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1cebf2b7d7b40dc85e05405ed6a8369" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 40067 } } peers { permanent_uuid: "6fedb9247f6a49c8b7f6a15745af0f60" member_type: VOTER last_known_addr { host: "127.16.43.189" port: 45261 } } peers { permanent_uuid: "ad96ce9a54a24a5e9a20d318025b3bb6" member_type: VOTER last_known_addr { host: "127.16.43.188" port: 41383 } } }
I20260620 09:28:39.226461 30099 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad96ce9a54a24a5e9a20d318025b3bb6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e1cebf2b7d7b40dc85e05405ed6a8369" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1cebf2b7d7b40dc85e05405ed6a8369" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 40067 } } peers { permanent_uuid: "6fedb9247f6a49c8b7f6a15745af0f60" member_type: VOTER last_known_addr { host: "127.16.43.189" port: 45261 } } peers { permanent_uuid: "ad96ce9a54a24a5e9a20d318025b3bb6" member_type: VOTER last_known_addr { host: "127.16.43.188" port: 41383 } } }
I20260620 09:28:39.226531 30118 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1cebf2b7d7b40dc85e05405ed6a8369 [sys.catalog]: This master's current role is: LEADER
I20260620 09:28:39.226536 30099 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad96ce9a54a24a5e9a20d318025b3bb6 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 09:28:39.226521 30134 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fedb9247f6a49c8b7f6a15745af0f60 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e1cebf2b7d7b40dc85e05405ed6a8369" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1cebf2b7d7b40dc85e05405ed6a8369" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 40067 } } peers { permanent_uuid: "6fedb9247f6a49c8b7f6a15745af0f60" member_type: VOTER last_known_addr { host: "127.16.43.189" port: 45261 } } peers { permanent_uuid: "ad96ce9a54a24a5e9a20d318025b3bb6" member_type: VOTER last_known_addr { host: "127.16.43.188" port: 41383 } } }
I20260620 09:28:39.226606 30134 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fedb9247f6a49c8b7f6a15745af0f60 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 09:28:39.226666 30149 catalog_manager.cc:1347] Generated new cluster ID: 1375dd1f9f48497d8c90465bc69116e5
I20260620 09:28:39.226703 30149 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:28:39.226804 30152 mvcc.cc:205] Tried to move back new op lower bound from 7298857857937178624 to 7298857857930317824. Current Snapshot: MvccSnapshot[applied={T|T < 7298857857937178624}]
I20260620 09:28:39.226954 30151 mvcc.cc:205] Tried to move back new op lower bound from 7298857857937178624 to 7298857857930317824. Current Snapshot: MvccSnapshot[applied={T|T < 7298857857937178624}]
I20260620 09:28:39.243954 30149 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:28:39.244659 30149 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:28:39.253449 30149 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e1cebf2b7d7b40dc85e05405ed6a8369: Generated new TSK 0
I20260620 09:28:39.253713 30149 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 09:28:39.980881 30110 catalog_manager.cc:1259] Loaded cluster ID: 1375dd1f9f48497d8c90465bc69116e5
I20260620 09:28:39.981027 30110 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ad96ce9a54a24a5e9a20d318025b3bb6: loading cluster ID for follower catalog manager: success
I20260620 09:28:39.981922 30110 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ad96ce9a54a24a5e9a20d318025b3bb6: acquiring CA information for follower catalog manager: success
I20260620 09:28:39.982266 30110 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ad96ce9a54a24a5e9a20d318025b3bb6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 09:28:40.034643 30145 catalog_manager.cc:1259] Loaded cluster ID: 1375dd1f9f48497d8c90465bc69116e5
I20260620 09:28:40.034729 30145 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6fedb9247f6a49c8b7f6a15745af0f60: loading cluster ID for follower catalog manager: success
I20260620 09:28:40.035730 30145 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6fedb9247f6a49c8b7f6a15745af0f60: acquiring CA information for follower catalog manager: success
I20260620 09:28:40.036067 30145 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6fedb9247f6a49c8b7f6a15745af0f60: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 09:28:40.323024 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 29874
I20260620 09:28:40.328496 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 29949
I20260620 09:28:40.333645 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 30024
2026-06-20T09:28:40Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3015 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781947562949448-16558-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:28:40 dist-test-slave-706q krb5kdc[30164](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:28:40 dist-test-slave-706q krb5kdc[30164](info): set up 2 sockets
Jun 20 09:28:40 dist-test-slave-706q krb5kdc[30164](info): commencing operation
krb5kdc: starting...
W20260620 09:28:41.428982 16558 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781947562949448-16558-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781947562949448-16558-0/krb5kdc/test-user.keytab.
Jun 20 09:28:41 dist-test-slave-706q krb5kdc[30164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947721, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:28:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:28:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.43.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.43.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.43.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.190.keytab.
Entry for principal kudu/127.16.43.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.190.keytab.
I20260620 09:28:41.551057 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781947562949448-16558-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781947562949448-16558-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781947562949448-16558-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.43.190:42113
--webserver_interface=127.16.43.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.190.keytab
--principal=kudu/127.16.43.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.43.148:34525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.43.190:42113
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781947562949448-16558-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781947562949448-16558-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781947562949448-16558-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:28:41.633318 30177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:28:41.633598 30177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:28:41.633641 30177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:28:41.634859 30177 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:28:41.634933 30177 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:28:41.634958 30177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:28:41.634979 30177 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:28:41.635005 30177 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:28:41.636607 30177 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:34525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781947562949448-16558-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781947562949448-16558-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.43.190:42113
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.190.keytab
--principal=kudu/127.16.43.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.43.190:42113
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.43.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30177
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781947562949448-16558-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:28:41.636907 30177 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:28:41.637149 30177 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:28:41.642402 30182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:28:41.642517 30177 server_base.cc:1034] running on GCE node
W20260620 09:28:41.642401 30183 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:28:41.642405 30185 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:28:41.642915 30177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:28:41.643266 30177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:28:41.644456 30177 hybrid_clock.cc:648] HybridClock initialized: now 1781947721644437 us; error 38 us; skew 500 ppm
Jun 20 09:28:41 dist-test-slave-706q krb5kdc[30164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947721, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:28:41.647285 30177 init.cc:377] Logged in from keytab as kudu/127.16.43.190@KRBTEST.COM (short username kudu)
I20260620 09:28:41.647843 30177 webserver.cc:460] Webserver started at http://127.16.43.190:42861/ using document root <none> and password file <none>
I20260620 09:28:41.648103 30177 fs_manager.cc:359] Metadata directory not provided
I20260620 09:28:41.648175 30177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:28:41.648319 30177 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:28:41.649412 30177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781947562949448-16558-0/minicluster-data/master-0/data/instance:
uuid: "fd0faa491ce14c86a6d3d8996163bde9"
format_stamp: "Formatted at 2026-06-20 09:28:41 on dist-test-slave-706q"
server_key: "0b1c65756877c37d8f36e5e61404de15"
server_key_iv: "287c9c04ee2c2bb4cbfb3f2193dbf464"
server_key_version: "encryptionkey@0"
I20260620 09:28:41.649788 30177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781947562949448-16558-0/minicluster-data/master-0/wal/instance:
uuid: "fd0faa491ce14c86a6d3d8996163bde9"
format_stamp: "Formatted at 2026-06-20 09:28:41 on dist-test-slave-706q"
server_key: "0b1c65756877c37d8f36e5e61404de15"
server_key_iv: "287c9c04ee2c2bb4cbfb3f2193dbf464"
server_key_version: "encryptionkey@0"
I20260620 09:28:41.651173 30177 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 09:28:41.651963 30192 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:28:41.652319 30177 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 09:28:41.652457 30177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781947562949448-16558-0/minicluster-data/master-0/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781947562949448-16558-0/minicluster-data/master-0/wal
uuid: "fd0faa491ce14c86a6d3d8996163bde9"
format_stamp: "Formatted at 2026-06-20 09:28:41 on dist-test-slave-706q"
server_key: "0b1c65756877c37d8f36e5e61404de15"
server_key_iv: "287c9c04ee2c2bb4cbfb3f2193dbf464"
server_key_version: "encryptionkey@0"
I20260620 09:28:41.652537 30177 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781947562949448-16558-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781947562949448-16558-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781947562949448-16558-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:28:41.664275 30177 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:28:41.666780 30177 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:28:41.666955 30177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:28:41.671257 30177 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.190:42113
I20260620 09:28:41.671316 30244 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.190:42113 every 8 connection(s)
I20260620 09:28:41.671772 30177 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
I20260620 09:28:41.672670 30245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:28:41.675582 30245 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd0faa491ce14c86a6d3d8996163bde9: Bootstrap starting.
I20260620 09:28:41.675900 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 30177
I20260620 09:28:41.676012 16558 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781947562949448-16558-0/minicluster-data/master-0/wal/instance
I20260620 09:28:41.676230 16558 external_mini_cluster.cc:1428] Setting key 21364f5f425de957a51ccfcc3e2ef43f
I20260620 09:28:41.676496 30245 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fd0faa491ce14c86a6d3d8996163bde9: Neither blocks nor log segments found. Creating new log.
I20260620 09:28:41.676827 30245 log.cc:824] T 00000000000000000000000000000000 P fd0faa491ce14c86a6d3d8996163bde9: Log is configured to *not* fsync() on all Append() calls
I20260620 09:28:41.677681 30245 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd0faa491ce14c86a6d3d8996163bde9: No bootstrap required, opened a new log
I20260620 09:28:41.680140 30245 raft_consensus.cc:348] T 00000000000000000000000000000000 P fd0faa491ce14c86a6d3d8996163bde9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd0faa491ce14c86a6d3d8996163bde9" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 42113 } }
I20260620 09:28:41.680335 30245 raft_consensus.cc:374] T 00000000000000000000000000000000 P fd0faa491ce14c86a6d3d8996163bde9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:28:41.680372 30245 raft_consensus.cc:729] T 00000000000000000000000000000000 P fd0faa491ce14c86a6d3d8996163bde9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd0faa491ce14c86a6d3d8996163bde9, State: Initialized, Role: FOLLOWER
I20260620 09:28:41.680500 30245 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd0faa491ce14c86a6d3d8996163bde9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd0faa491ce14c86a6d3d8996163bde9" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 42113 } }
I20260620 09:28:41.680579 30245 raft_consensus.cc:388] T 00000000000000000000000000000000 P fd0faa491ce14c86a6d3d8996163bde9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:28:41.680609 30245 raft_consensus.cc:482] T 00000000000000000000000000000000 P fd0faa491ce14c86a6d3d8996163bde9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:28:41.680653 30245 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fd0faa491ce14c86a6d3d8996163bde9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:28:41.681320 30245 raft_consensus.cc:504] T 00000000000000000000000000000000 P fd0faa491ce14c86a6d3d8996163bde9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd0faa491ce14c86a6d3d8996163bde9" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 42113 } }
I20260620 09:28:41.681455 30245 leader_election.cc:302] T 00000000000000000000000000000000 P fd0faa491ce14c86a6d3d8996163bde9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fd0faa491ce14c86a6d3d8996163bde9; no voters:
I20260620 09:28:41.681651 30245 leader_election.cc:288] T 00000000000000000000000000000000 P fd0faa491ce14c86a6d3d8996163bde9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:28:41.681692 30250 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fd0faa491ce14c86a6d3d8996163bde9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:28:41.681861 30250 raft_consensus.cc:686] T 00000000000000000000000000000000 P fd0faa491ce14c86a6d3d8996163bde9 [term 1 LEADER]: Becoming Leader. State: Replica: fd0faa491ce14c86a6d3d8996163bde9, State: Running, Role: LEADER
I20260620 09:28:41.681979 30245 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd0faa491ce14c86a6d3d8996163bde9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:28:41.682034 30250 consensus_queue.cc:237] T 00000000000000000000000000000000 P fd0faa491ce14c86a6d3d8996163bde9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd0faa491ce14c86a6d3d8996163bde9" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 42113 } }
I20260620 09:28:41.683298 30252 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd0faa491ce14c86a6d3d8996163bde9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd0faa491ce14c86a6d3d8996163bde9. Latest consensus state: current_term: 1 leader_uuid: "fd0faa491ce14c86a6d3d8996163bde9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd0faa491ce14c86a6d3d8996163bde9" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 42113 } } }
I20260620 09:28:41.683386 30252 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd0faa491ce14c86a6d3d8996163bde9 [sys.catalog]: This master's current role is: LEADER
I20260620 09:28:41.683399 30251 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd0faa491ce14c86a6d3d8996163bde9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fd0faa491ce14c86a6d3d8996163bde9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd0faa491ce14c86a6d3d8996163bde9" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 42113 } } }
I20260620 09:28:41.683481 30251 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd0faa491ce14c86a6d3d8996163bde9 [sys.catalog]: This master's current role is: LEADER
Jun 20 09:28:41 dist-test-slave-706q krb5kdc[30164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947721, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.43.190@KRBTEST.COM
W20260620 09:28:41.684015 30265 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fd0faa491ce14c86a6d3d8996163bde9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 09:28:41.684055 30265 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 09:28:41.684128 30266 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:28:41.684307 30266 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:28:41.686110 30266 catalog_manager.cc:1347] Generated new cluster ID: 96f5d1771aba4e5fa66cda3e738430a4
I20260620 09:28:41.686175 30266 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:28:41.688584 30248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:41.676886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60302 (local address 127.16.43.190:42113)
0620 09:28:41.677058 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 09:28:41.677068 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 09:28:41.677143 (+ 75us) server_negotiation.cc:407] Connection header received
0620 09:28:41.677697 (+ 554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:28:41.677706 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:28:41.677761 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:28:41.677934 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:28:41.678768 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:28:41.679854 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:28:41.680906 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:28:41.681322 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:28:41.681532 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:28:41.684587 (+ 3055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:28:41.684592 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:28:41.684592 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:28:41.684605 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:28:41.686620 (+ 2015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:28:41.687154 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:28:41.687157 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:28:41.687158 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:28:41.687269 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:28:41.687590 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:28:41.687591 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:28:41.687591 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:28:41.687917 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:28:41.688107 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0620 09:28:41.688223 (+ 116us) server_negotiation.cc:299] Negotiation successful
0620 09:28:41.688381 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":77,"threads_started":1}
I20260620 09:28:41.689680 16558 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 09:28:41.701161 30266 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:28:41.701894 30266 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:28:41.706152 30266 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fd0faa491ce14c86a6d3d8996163bde9: Generated new TSK 0
I20260620 09:28:41.706355 30266 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 09:28:41.743686 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 30177
2026-06-20T09:28:41Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1407 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:28:41 dist-test-slave-706q krb5kdc[30272](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:28:41 dist-test-slave-706q krb5kdc[30272](info): set up 2 sockets
Jun 20 09:28:41 dist-test-slave-706q krb5kdc[30272](info): commencing operation
krb5kdc: starting...
W20260620 09:28:42.800635 16558 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/krb5kdc/test-user.keytab.
Jun 20 09:28:42 dist-test-slave-706q krb5kdc[30272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947722, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:28:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:28:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.43.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.43.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.43.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.190.keytab.
Entry for principal kudu/127.16.43.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.190.keytab.
I20260620 09:28:42.927629 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.43.190:38073
--webserver_interface=127.16.43.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.190.keytab
--principal=kudu/127.16.43.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.43.148:42633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.43.190:38073
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:28:43.010560 30285 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:28:43.010793 30285 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:28:43.010818 30285 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:28:43.012002 30285 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:28:43.012040 30285 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:28:43.012054 30285 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:28:43.012068 30285 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:28:43.012086 30285 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:28:43.013520 30285 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:42633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.43.190:38073
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.190.keytab
--principal=kudu/127.16.43.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.43.190:38073
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.43.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30285
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:28:43.013723 30285 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:28:43.013906 30285 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:28:43.018977 30291 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:28:43.019011 30293 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:28:43.019011 30285 server_base.cc:1034] running on GCE node
W20260620 09:28:43.018977 30290 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:28:43.019455 30285 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:28:43.019838 30285 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:28:43.020987 30285 hybrid_clock.cc:648] HybridClock initialized: now 1781947723020967 us; error 38 us; skew 500 ppm
Jun 20 09:28:43 dist-test-slave-706q krb5kdc[30272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947723, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:28:43.029026 30285 init.cc:377] Logged in from keytab as kudu/127.16.43.190@KRBTEST.COM (short username kudu)
I20260620 09:28:43.029539 30285 webserver.cc:460] Webserver started at http://127.16.43.190:43503/ using document root <none> and password file <none>
I20260620 09:28:43.029749 30285 fs_manager.cc:359] Metadata directory not provided
I20260620 09:28:43.029793 30285 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:28:43.029887 30285 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:28:43.031024 30285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/master-0/data/instance:
uuid: "516727c9072f4e0f8034238b71a751d3"
format_stamp: "Formatted at 2026-06-20 09:28:43 on dist-test-slave-706q"
server_key: "1bf6a9adabdb7e214fbfe067da2bb3f6"
server_key_iv: "bc521b0d457322537acc73bfbfeec5ae"
server_key_version: "encryptionkey@0"
I20260620 09:28:43.031364 30285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/master-0/wal/instance:
uuid: "516727c9072f4e0f8034238b71a751d3"
format_stamp: "Formatted at 2026-06-20 09:28:43 on dist-test-slave-706q"
server_key: "1bf6a9adabdb7e214fbfe067da2bb3f6"
server_key_iv: "bc521b0d457322537acc73bfbfeec5ae"
server_key_version: "encryptionkey@0"
I20260620 09:28:43.032853 30285 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 09:28:43.033568 30300 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:28:43.033735 30285 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 09:28:43.033820 30285 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/master-0/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/master-0/wal
uuid: "516727c9072f4e0f8034238b71a751d3"
format_stamp: "Formatted at 2026-06-20 09:28:43 on dist-test-slave-706q"
server_key: "1bf6a9adabdb7e214fbfe067da2bb3f6"
server_key_iv: "bc521b0d457322537acc73bfbfeec5ae"
server_key_version: "encryptionkey@0"
I20260620 09:28:43.033877 30285 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:28:43.045680 30285 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:28:43.047969 30285 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:28:43.048095 30285 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:28:43.052024 30285 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.190:38073
I20260620 09:28:43.052094 30352 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.190:38073 every 8 connection(s)
I20260620 09:28:43.052424 30285 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
I20260620 09:28:43.052556 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 30285
I20260620 09:28:43.052685 16558 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/master-0/wal/instance
I20260620 09:28:43.052882 16558 external_mini_cluster.cc:1428] Setting key 31dc838781f1540b6595ca4df00199dc
I20260620 09:28:43.053555 30353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:28:43.056123 30353 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 516727c9072f4e0f8034238b71a751d3: Bootstrap starting.
I20260620 09:28:43.056726 30353 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 516727c9072f4e0f8034238b71a751d3: Neither blocks nor log segments found. Creating new log.
I20260620 09:28:43.057001 30353 log.cc:824] T 00000000000000000000000000000000 P 516727c9072f4e0f8034238b71a751d3: Log is configured to *not* fsync() on all Append() calls
I20260620 09:28:43.057811 30353 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 516727c9072f4e0f8034238b71a751d3: No bootstrap required, opened a new log
Jun 20 09:28:43 dist-test-slave-706q krb5kdc[30272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947722, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.43.190@KRBTEST.COM
I20260620 09:28:43.060171 30353 raft_consensus.cc:348] T 00000000000000000000000000000000 P 516727c9072f4e0f8034238b71a751d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "516727c9072f4e0f8034238b71a751d3" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 38073 } }
I20260620 09:28:43.060357 30353 raft_consensus.cc:374] T 00000000000000000000000000000000 P 516727c9072f4e0f8034238b71a751d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:28:43.060401 30353 raft_consensus.cc:729] T 00000000000000000000000000000000 P 516727c9072f4e0f8034238b71a751d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 516727c9072f4e0f8034238b71a751d3, State: Initialized, Role: FOLLOWER
I20260620 09:28:43.060565 30353 consensus_queue.cc:260] T 00000000000000000000000000000000 P 516727c9072f4e0f8034238b71a751d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "516727c9072f4e0f8034238b71a751d3" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 38073 } }
I20260620 09:28:43.060689 30353 raft_consensus.cc:388] T 00000000000000000000000000000000 P 516727c9072f4e0f8034238b71a751d3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:28:43.060724 30353 raft_consensus.cc:482] T 00000000000000000000000000000000 P 516727c9072f4e0f8034238b71a751d3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:28:43.060771 30353 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 516727c9072f4e0f8034238b71a751d3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:28:43.061303 30353 raft_consensus.cc:504] T 00000000000000000000000000000000 P 516727c9072f4e0f8034238b71a751d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "516727c9072f4e0f8034238b71a751d3" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 38073 } }
I20260620 09:28:43.061411 30353 leader_election.cc:302] T 00000000000000000000000000000000 P 516727c9072f4e0f8034238b71a751d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 516727c9072f4e0f8034238b71a751d3; no voters:
I20260620 09:28:43.061581 30353 leader_election.cc:288] T 00000000000000000000000000000000 P 516727c9072f4e0f8034238b71a751d3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:28:43.061635 30358 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 516727c9072f4e0f8034238b71a751d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:28:43.061815 30358 raft_consensus.cc:686] T 00000000000000000000000000000000 P 516727c9072f4e0f8034238b71a751d3 [term 1 LEADER]: Becoming Leader. State: Replica: 516727c9072f4e0f8034238b71a751d3, State: Running, Role: LEADER
I20260620 09:28:43.061848 30353 sys_catalog.cc:564] T 00000000000000000000000000000000 P 516727c9072f4e0f8034238b71a751d3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:28:43.061981 30358 consensus_queue.cc:237] T 00000000000000000000000000000000 P 516727c9072f4e0f8034238b71a751d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "516727c9072f4e0f8034238b71a751d3" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 38073 } }
I20260620 09:28:43.063381 30360 sys_catalog.cc:455] T 00000000000000000000000000000000 P 516727c9072f4e0f8034238b71a751d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 516727c9072f4e0f8034238b71a751d3. Latest consensus state: current_term: 1 leader_uuid: "516727c9072f4e0f8034238b71a751d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "516727c9072f4e0f8034238b71a751d3" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 38073 } } }
I20260620 09:28:43.063429 30359 sys_catalog.cc:455] T 00000000000000000000000000000000 P 516727c9072f4e0f8034238b71a751d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "516727c9072f4e0f8034238b71a751d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "516727c9072f4e0f8034238b71a751d3" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 38073 } } }
I20260620 09:28:43.063558 30360 sys_catalog.cc:458] T 00000000000000000000000000000000 P 516727c9072f4e0f8034238b71a751d3 [sys.catalog]: This master's current role is: LEADER
I20260620 09:28:43.063616 30359 sys_catalog.cc:458] T 00000000000000000000000000000000 P 516727c9072f4e0f8034238b71a751d3 [sys.catalog]: This master's current role is: LEADER
W20260620 09:28:43.064009 30373 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 516727c9072f4e0f8034238b71a751d3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 09:28:43.064128 30373 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 09:28:43.064515 30356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:43.053427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34458 (local address 127.16.43.190:38073)
0620 09:28:43.053622 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 09:28:43.053633 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 09:28:43.053666 (+ 33us) server_negotiation.cc:407] Connection header received
0620 09:28:43.054126 (+ 460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:28:43.054132 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:28:43.054179 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:28:43.054329 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:28:43.054953 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:28:43.055938 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:28:43.056742 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:28:43.057261 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:28:43.057369 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:28:43.060083 (+ 2714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:28:43.060088 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:28:43.060089 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:28:43.060102 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:28:43.062505 (+ 2403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:28:43.062996 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:28:43.062999 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:28:43.062999 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:28:43.063122 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:28:43.063413 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:28:43.063414 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:28:43.063415 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:28:43.063832 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:28:43.063981 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 09:28:43.064153 (+ 172us) server_negotiation.cc:299] Negotiation successful
0620 09:28:43.064309 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":91,"threads_started":1}
I20260620 09:28:43.067698 30375 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:28:43.067929 30375 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:28:43.069649 30375 catalog_manager.cc:1347] Generated new cluster ID: 8b0481e1fb5546209dc536267159980d
I20260620 09:28:43.069713 30375 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:28:43.084635 30375 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:28:43.085386 30375 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:28:43.088842 30375 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 516727c9072f4e0f8034238b71a751d3: Generated new TSK 0
I20260620 09:28:43.089082 30375 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.43.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.43.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.43.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.129.keytab.
Entry for principal kudu/127.16.43.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.129.keytab.
I20260620 09:28:43.120807 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.43.129:0
--local_ip_for_outbound_sockets=127.16.43.129
--webserver_interface=127.16.43.129
--webserver_port=0
--tserver_master_addrs=127.16.43.190:38073
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.129.keytab
--principal=kudu/127.16.43.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.43.148:42633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:28:43.204777 30379 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:28:43.205001 30379 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:28:43.205026 30379 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:28:43.206228 30379 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:28:43.206275 30379 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:28:43.206303 30379 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.43.129
I20260620 09:28:43.207806 30379 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:42633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.129.keytab
--principal=kudu/127.16.43.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.43.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.43.129
--webserver_port=0
--tserver_master_addrs=127.16.43.190:38073
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30379
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.43.129
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:28:43.208003 30379 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:28:43.208206 30379 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:28:43.213292 30384 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:28:43.213352 30385 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:28:43.213289 30387 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:28:43.213331 30379 server_base.cc:1034] running on GCE node
I20260620 09:28:43.213752 30379 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:28:43.214004 30379 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:28:43.215181 30379 hybrid_clock.cc:648] HybridClock initialized: now 1781947723215154 us; error 40 us; skew 500 ppm
Jun 20 09:28:43 dist-test-slave-706q krb5kdc[30272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947723, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:28:43.217870 30379 init.cc:377] Logged in from keytab as kudu/127.16.43.129@KRBTEST.COM (short username kudu)
I20260620 09:28:43.218324 30379 webserver.cc:460] Webserver started at http://127.16.43.129:42289/ using document root <none> and password file <none>
I20260620 09:28:43.218509 30379 fs_manager.cc:359] Metadata directory not provided
I20260620 09:28:43.218549 30379 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:28:43.218642 30379 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:28:43.219715 30379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-0/data/instance:
uuid: "55de9465487443d0bcd713075a4d8438"
format_stamp: "Formatted at 2026-06-20 09:28:43 on dist-test-slave-706q"
server_key: "b8cf4182d8252833daa52bc821848630"
server_key_iv: "1d181cee9ea91d529ae9d23154d68401"
server_key_version: "encryptionkey@0"
I20260620 09:28:43.220017 30379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-0/wal/instance:
uuid: "55de9465487443d0bcd713075a4d8438"
format_stamp: "Formatted at 2026-06-20 09:28:43 on dist-test-slave-706q"
server_key: "b8cf4182d8252833daa52bc821848630"
server_key_iv: "1d181cee9ea91d529ae9d23154d68401"
server_key_version: "encryptionkey@0"
I20260620 09:28:43.221275 30379 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:28:43.221982 30394 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:28:43.222172 30379 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 09:28:43.222252 30379 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-0/wal
uuid: "55de9465487443d0bcd713075a4d8438"
format_stamp: "Formatted at 2026-06-20 09:28:43 on dist-test-slave-706q"
server_key: "b8cf4182d8252833daa52bc821848630"
server_key_iv: "1d181cee9ea91d529ae9d23154d68401"
server_key_version: "encryptionkey@0"
I20260620 09:28:43.222308 30379 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:28:43.247831 30379 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:28:43.250152 30379 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:28:43.250272 30379 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:28:43.250465 30379 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:28:43.250816 30379 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:28:43.250850 30379 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:28:43.250897 30379 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:28:43.250923 30379 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:28:43.256789 30379 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.129:43333
I20260620 09:28:43.256871 30507 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.129:43333 every 8 connection(s)
I20260620 09:28:43.257210 30379 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-0/data/info.pb
Jun 20 09:28:43 dist-test-slave-706q krb5kdc[30272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947723, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.129@KRBTEST.COM for kudu/127.16.43.190@KRBTEST.COM
I20260620 09:28:43.266167 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 30379
I20260620 09:28:43.266283 16558 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-0/wal/instance
I20260620 09:28:43.266474 16558 external_mini_cluster.cc:1428] Setting key 92e56ba8f20f0219f08f01e20baeac1a
I20260620 09:28:43.266783 30356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:43.258002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.129:60637 (local address 127.16.43.190:38073)
0620 09:28:43.258051 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 09:28:43.258053 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:28:43.258247 (+ 194us) server_negotiation.cc:407] Connection header received
0620 09:28:43.258794 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:28:43.258796 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:28:43.258809 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:28:43.258864 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:28:43.259894 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:28:43.260417 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:28:43.261464 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:28:43.261761 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:28:43.261826 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:28:43.264147 (+ 2321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:28:43.264148 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:28:43.264148 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:28:43.264155 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:28:43.265388 (+ 1233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:28:43.265759 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:28:43.265760 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:28:43.265761 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:28:43.265811 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:28:43.266034 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:28:43.266036 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:28:43.266036 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:28:43.266194 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:28:43.266278 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 09:28:43.266584 (+ 306us) server_negotiation.cc:299] Negotiation successful
0620 09:28:43.266683 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 09:28:43.266927 30510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:43.257960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:38073 (local address 127.16.43.129:60637)
0620 09:28:43.258111 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 09:28:43.258153 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 09:28:43.258683 (+ 530us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:28:43.258938 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:28:43.258942 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:28:43.259593 (+ 651us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:28:43.259794 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:28:43.259799 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:28:43.260514 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:28:43.260515 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:28:43.261346 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:28:43.261348 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:28:43.261840 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:28:43.261841 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:28:43.261947 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:28:43.262502 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:28:43.262512 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:28:43.264063 (+ 1551us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:28:43.265457 (+ 1394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:28:43.265463 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:28:43.265465 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:28:43.265697 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:28:43.265865 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:28:43.265865 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:28:43.265865 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:28:43.265964 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:28:43.266316 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:28:43.266322 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:28:43.266449 (+ 127us) client_negotiation.cc:769] Sending connection context
0620 09:28:43.266536 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 09:28:43.266693 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":62,"threads_started":1}
I20260620 09:28:43.267515 30508 heartbeater.cc:347] Connected to a master server at 127.16.43.190:38073
I20260620 09:28:43.267663 30508 heartbeater.cc:464] Registering TS with master...
I20260620 09:28:43.267844 30508 heartbeater.cc:511] Master 127.16.43.190:38073 requested a full tablet report, sending...
I20260620 09:28:43.268414 30317 ts_manager.cc:194] Registered new tserver with Master: 55de9465487443d0bcd713075a4d8438 (127.16.43.129:43333)
I20260620 09:28:43.269364 30317 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.43.129@KRBTEST.COM'} at 127.16.43.129:60637
WARNING: no policy specified for kudu/127.16.43.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.43.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.43.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.130.keytab.
Entry for principal kudu/127.16.43.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.130.keytab.
I20260620 09:28:43.294934 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.43.130:0
--local_ip_for_outbound_sockets=127.16.43.130
--webserver_interface=127.16.43.130
--webserver_port=0
--tserver_master_addrs=127.16.43.190:38073
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.130.keytab
--principal=kudu/127.16.43.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.43.148:42633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:28:43.376755 30513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:28:43.376973 30513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:28:43.376998 30513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:28:43.378253 30513 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:28:43.378305 30513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:28:43.378333 30513 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.43.130
I20260620 09:28:43.379899 30513 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:42633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.130.keytab
--principal=kudu/127.16.43.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.43.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.43.130
--webserver_port=0
--tserver_master_addrs=127.16.43.190:38073
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30513
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.43.130
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:28:43.380180 30513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:28:43.380429 30513 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:28:43.385708 30518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:28:43.385773 30519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:28:43.385748 30521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:28:43.385861 30513 server_base.cc:1034] running on GCE node
I20260620 09:28:43.386174 30513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:28:43.386459 30513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:28:43.387586 30513 hybrid_clock.cc:648] HybridClock initialized: now 1781947723387566 us; error 47 us; skew 500 ppm
Jun 20 09:28:43 dist-test-slave-706q krb5kdc[30272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947723, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:28:43.390252 30513 init.cc:377] Logged in from keytab as kudu/127.16.43.130@KRBTEST.COM (short username kudu)
I20260620 09:28:43.390679 30513 webserver.cc:460] Webserver started at http://127.16.43.130:36893/ using document root <none> and password file <none>
I20260620 09:28:43.390946 30513 fs_manager.cc:359] Metadata directory not provided
I20260620 09:28:43.391016 30513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:28:43.391165 30513 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:28:43.392321 30513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-1/data/instance:
uuid: "52358e0cc7da4808a747d83dab337afd"
format_stamp: "Formatted at 2026-06-20 09:28:43 on dist-test-slave-706q"
server_key: "837d3a2056f4b79c44161b683e4c2532"
server_key_iv: "aceb559cad56c1fd490b5a56da04005b"
server_key_version: "encryptionkey@0"
I20260620 09:28:43.392691 30513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-1/wal/instance:
uuid: "52358e0cc7da4808a747d83dab337afd"
format_stamp: "Formatted at 2026-06-20 09:28:43 on dist-test-slave-706q"
server_key: "837d3a2056f4b79c44161b683e4c2532"
server_key_iv: "aceb559cad56c1fd490b5a56da04005b"
server_key_version: "encryptionkey@0"
I20260620 09:28:43.394052 30513 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 09:28:43.394870 30528 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:28:43.395061 30513 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:28:43.395171 30513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-1/wal
uuid: "52358e0cc7da4808a747d83dab337afd"
format_stamp: "Formatted at 2026-06-20 09:28:43 on dist-test-slave-706q"
server_key: "837d3a2056f4b79c44161b683e4c2532"
server_key_iv: "aceb559cad56c1fd490b5a56da04005b"
server_key_version: "encryptionkey@0"
I20260620 09:28:43.395253 30513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:28:43.436595 30513 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:28:43.439026 30513 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:28:43.439203 30513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:28:43.439461 30513 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:28:43.439817 30513 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:28:43.439869 30513 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:28:43.439913 30513 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:28:43.439951 30513 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:28:43.446266 30513 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.130:38497
I20260620 09:28:43.446312 30641 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.130:38497 every 8 connection(s)
I20260620 09:28:43.446650 30513 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-1/data/info.pb
I20260620 09:28:43.450493 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 30513
I20260620 09:28:43.450604 16558 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-1/wal/instance
I20260620 09:28:43.450783 16558 external_mini_cluster.cc:1428] Setting key a957100a7cde9db66e3c314214660f18
Jun 20 09:28:43 dist-test-slave-706q krb5kdc[30272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947723, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.130@KRBTEST.COM for kudu/127.16.43.190@KRBTEST.COM
I20260620 09:28:43.458082 30356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:43.447418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.130:47875 (local address 127.16.43.190:38073)
0620 09:28:43.447489 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0620 09:28:43.447491 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:28:43.447805 (+ 314us) server_negotiation.cc:407] Connection header received
0620 09:28:43.448292 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:28:43.448293 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:28:43.448303 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:28:43.448372 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:28:43.449316 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:28:43.450241 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:28:43.452400 (+ 2159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:28:43.452604 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:28:43.452668 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:28:43.455161 (+ 2493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:28:43.455163 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:28:43.455163 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:28:43.455171 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:28:43.456647 (+ 1476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:28:43.457102 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:28:43.457103 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:28:43.457104 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:28:43.457165 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:28:43.457411 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:28:43.457420 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:28:43.457420 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:28:43.457543 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:28:43.457617 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 09:28:43.457900 (+ 283us) server_negotiation.cc:299] Negotiation successful
0620 09:28:43.457984 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 09:28:43.458293 30644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:43.447405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:38073 (local address 127.16.43.130:47875)
0620 09:28:43.447692 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0620 09:28:43.447728 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 09:28:43.448210 (+ 482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:28:43.448429 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:28:43.448433 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:28:43.449006 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:28:43.449210 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:28:43.449216 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:28:43.450973 (+ 1757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:28:43.450974 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:28:43.452294 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:28:43.452297 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:28:43.452693 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:28:43.452695 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:28:43.452863 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:28:43.453515 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:28:43.453524 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:28:43.455047 (+ 1523us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:28:43.456753 (+ 1706us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:28:43.456762 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:28:43.456764 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:28:43.457009 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:28:43.457230 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:28:43.457230 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:28:43.457231 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:28:43.457344 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:28:43.457607 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:28:43.457614 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:28:43.457760 (+ 146us) client_negotiation.cc:769] Sending connection context
0620 09:28:43.457866 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 09:28:43.458048 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":79,"threads_started":1}
I20260620 09:28:43.458781 30642 heartbeater.cc:347] Connected to a master server at 127.16.43.190:38073
WARNING: no policy specified for kudu/127.16.43.131@KRBTEST.COM; defaulting to no policy
I20260620 09:28:43.458884 30642 heartbeater.cc:464] Registering TS with master...
I20260620 09:28:43.459089 30642 heartbeater.cc:511] Master 127.16.43.190:38073 requested a full tablet report, sending...
I20260620 09:28:43.459487 30317 ts_manager.cc:194] Registered new tserver with Master: 52358e0cc7da4808a747d83dab337afd (127.16.43.130:38497)
I20260620 09:28:43.459980 30317 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.43.130@KRBTEST.COM'} at 127.16.43.130:47875
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.43.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.43.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.131.keytab.
Entry for principal kudu/127.16.43.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.131.keytab.
I20260620 09:28:43.478482 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.43.131:0
--local_ip_for_outbound_sockets=127.16.43.131
--webserver_interface=127.16.43.131
--webserver_port=0
--tserver_master_addrs=127.16.43.190:38073
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.131.keytab
--principal=kudu/127.16.43.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.43.148:42633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:28:43.560912 30647 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:28:43.561167 30647 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:28:43.561208 30647 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:28:43.562623 30647 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:28:43.562685 30647 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:28:43.562732 30647 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.43.131
I20260620 09:28:43.564277 30647 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:42633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.131.keytab
--principal=kudu/127.16.43.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.43.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.43.131
--webserver_port=0
--tserver_master_addrs=127.16.43.190:38073
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30647
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.43.131
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:28:43.564550 30647 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:28:43.564795 30647 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:28:43.569945 30655 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:28:43.569974 30652 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:28:43.569945 30653 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:28:43.570254 30647 server_base.cc:1034] running on GCE node
I20260620 09:28:43.570488 30647 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:28:43.570776 30647 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:28:43.571930 30647 hybrid_clock.cc:648] HybridClock initialized: now 1781947723571910 us; error 39 us; skew 500 ppm
Jun 20 09:28:43 dist-test-slave-706q krb5kdc[30272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947723, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:28:43.574397 30647 init.cc:377] Logged in from keytab as kudu/127.16.43.131@KRBTEST.COM (short username kudu)
I20260620 09:28:43.574851 30647 webserver.cc:460] Webserver started at http://127.16.43.131:44601/ using document root <none> and password file <none>
I20260620 09:28:43.575076 30647 fs_manager.cc:359] Metadata directory not provided
I20260620 09:28:43.575150 30647 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:28:43.575295 30647 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:28:43.576454 30647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-2/data/instance:
uuid: "ca9f7e88ebe741259e36b11e1ddedca5"
format_stamp: "Formatted at 2026-06-20 09:28:43 on dist-test-slave-706q"
server_key: "e2bf67ce12b0713de4b59bdf0f2819d3"
server_key_iv: "afc58e6e5dd41fb8d0402815203423de"
server_key_version: "encryptionkey@0"
I20260620 09:28:43.576838 30647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-2/wal/instance:
uuid: "ca9f7e88ebe741259e36b11e1ddedca5"
format_stamp: "Formatted at 2026-06-20 09:28:43 on dist-test-slave-706q"
server_key: "e2bf67ce12b0713de4b59bdf0f2819d3"
server_key_iv: "afc58e6e5dd41fb8d0402815203423de"
server_key_version: "encryptionkey@0"
I20260620 09:28:43.578205 30647 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 09:28:43.578920 30662 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:28:43.579155 30647 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:28:43.579245 30647 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-2/wal
uuid: "ca9f7e88ebe741259e36b11e1ddedca5"
format_stamp: "Formatted at 2026-06-20 09:28:43 on dist-test-slave-706q"
server_key: "e2bf67ce12b0713de4b59bdf0f2819d3"
server_key_iv: "afc58e6e5dd41fb8d0402815203423de"
server_key_version: "encryptionkey@0"
I20260620 09:28:43.579324 30647 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:28:43.597239 30647 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:28:43.599752 30647 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:28:43.599915 30647 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:28:43.600162 30647 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:28:43.600497 30647 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:28:43.600549 30647 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:28:43.600593 30647 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:28:43.600631 30647 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:28:43.606480 30647 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.131:33529
I20260620 09:28:43.606567 30775 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.131:33529 every 8 connection(s)
I20260620 09:28:43.606895 30647 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-2/data/info.pb
Jun 20 09:28:43 dist-test-slave-706q krb5kdc[30272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947723, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.131@KRBTEST.COM for kudu/127.16.43.190@KRBTEST.COM
I20260620 09:28:43.613627 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 30647
I20260620 09:28:43.613718 16558 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-2/wal/instance
I20260620 09:28:43.613868 16558 external_mini_cluster.cc:1428] Setting key c8954de4389a5b17ce9fb1f5250233f9
I20260620 09:28:43.616869 30356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:43.607654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.131:60925 (local address 127.16.43.190:38073)
0620 09:28:43.607718 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 09:28:43.607719 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 09:28:43.607914 (+ 195us) server_negotiation.cc:407] Connection header received
0620 09:28:43.608442 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:28:43.608443 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:28:43.608455 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:28:43.608518 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:28:43.609435 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:28:43.609961 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:28:43.610982 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:28:43.611173 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:28:43.611231 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:28:43.613561 (+ 2330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:28:43.613562 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:28:43.613562 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:28:43.613575 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:28:43.615383 (+ 1808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:28:43.615913 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:28:43.615914 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:28:43.615914 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:28:43.615981 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:28:43.616268 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:28:43.616270 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:28:43.616270 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:28:43.616403 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:28:43.616458 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 09:28:43.616716 (+ 258us) server_negotiation.cc:299] Negotiation successful
0620 09:28:43.616794 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 09:28:43.617026 30778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:43.607659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:38073 (local address 127.16.43.131:60925)
0620 09:28:43.607804 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 09:28:43.607840 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 09:28:43.608345 (+ 505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:28:43.608588 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:28:43.608591 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:28:43.609160 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:28:43.609350 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:28:43.609354 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:28:43.610053 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:28:43.610054 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:28:43.610892 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:28:43.610894 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:28:43.611267 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:28:43.611268 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:28:43.611381 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:28:43.611895 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:28:43.611907 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:28:43.613336 (+ 1429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:28:43.615500 (+ 2164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:28:43.615509 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:28:43.615511 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:28:43.615821 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:28:43.616046 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:28:43.616046 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:28:43.616047 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:28:43.616177 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:28:43.616458 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:28:43.616462 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:28:43.616585 (+ 123us) client_negotiation.cc:769] Sending connection context
0620 09:28:43.616674 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 09:28:43.616802 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":78,"threads_started":1}
I20260620 09:28:43.617483 30776 heartbeater.cc:347] Connected to a master server at 127.16.43.190:38073
I20260620 09:28:43.617621 30776 heartbeater.cc:464] Registering TS with master...
I20260620 09:28:43.617811 30776 heartbeater.cc:511] Master 127.16.43.190:38073 requested a full tablet report, sending...
I20260620 09:28:43.618186 30317 ts_manager.cc:194] Registered new tserver with Master: ca9f7e88ebe741259e36b11e1ddedca5 (127.16.43.131:33529)
I20260620 09:28:43.618551 30317 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.43.131@KRBTEST.COM'} at 127.16.43.131:60925
I20260620 09:28:43.625876 16558 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 09:28:43.632829 30356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:43.627122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34462 (local address 127.16.43.190:38073)
0620 09:28:43.627192 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0620 09:28:43.627194 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:28:43.627295 (+ 101us) server_negotiation.cc:407] Connection header received
0620 09:28:43.627338 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:28:43.627339 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:28:43.627347 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:28:43.627398 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:28:43.627994 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:28:43.628466 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:28:43.629144 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:28:43.629340 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:28:43.629393 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:28:43.630528 (+ 1135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:28:43.630529 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:28:43.630529 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:28:43.630535 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:28:43.631873 (+ 1338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:28:43.632192 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:28:43.632193 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:28:43.632193 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:28:43.632243 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:28:43.632396 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:28:43.632396 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:28:43.632397 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:28:43.632534 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:28:43.632628 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 09:28:43.632703 (+ 75us) server_negotiation.cc:299] Negotiation successful
0620 09:28:43.632762 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 09:28:43.634377 30317 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34462:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 09:28:43.636056 30317 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 09:28:43.642724 30790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:43.638485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49074 (local address 127.16.43.130:38497)
0620 09:28:43.638978 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0620 09:28:43.638983 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:28:43.638992 (+ 9us) server_negotiation.cc:407] Connection header received
0620 09:28:43.639027 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:28:43.639029 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:28:43.639044 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:28:43.639141 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:28:43.639400 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:28:43.640234 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:28:43.641708 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:28:43.642290 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:28:43.642378 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:28:43.642442 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 09:28:43.642590 (+ 148us) server_negotiation.cc:299] Negotiation successful
0620 09:28:43.642634 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":89,"threads_started":1}
I20260620 09:28:43.642714 30791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:43.638383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.130:38497 (local address 127.0.0.1:49074)
0620 09:28:43.638712 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0620 09:28:43.638756 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 09:28:43.638823 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:28:43.639223 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:28:43.639224 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:28:43.639226 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:28:43.639282 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:28:43.639284 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:28:43.640330 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:28:43.640332 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:28:43.641616 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:28:43.641619 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:28:43.642377 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:28:43.642378 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:28:43.642459 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:28:43.642473 (+ 14us) client_negotiation.cc:769] Sending connection context
0620 09:28:43.642570 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 09:28:43.642634 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":51,"threads_started":1}
I20260620 09:28:43.642930 30788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:43.638585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.131:33529 (local address 127.0.0.1:57336)
0620 09:28:43.639380 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0620 09:28:43.639390 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 09:28:43.639439 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:28:43.639616 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:28:43.639617 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:28:43.639620 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:28:43.639660 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:28:43.639661 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:28:43.640794 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:28:43.640795 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:28:43.641740 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:28:43.641743 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:28:43.642672 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:28:43.642673 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:28:43.642744 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:28:43.642748 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 09:28:43.642798 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 09:28:43.642858 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":777,"mutex_wait_us":2,"thread_start_us":59,"threads_started":1}
I20260620 09:28:43.642992 30789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:43.638491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43982 (local address 127.16.43.129:43333)
0620 09:28:43.638710 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0620 09:28:43.638716 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 09:28:43.638918 (+ 202us) server_negotiation.cc:407] Connection header received
0620 09:28:43.638968 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:28:43.638971 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:28:43.638989 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:28:43.639071 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:28:43.639258 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:28:43.640443 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:28:43.641743 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:28:43.642501 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:28:43.642576 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:28:43.642635 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 09:28:43.642856 (+ 221us) server_negotiation.cc:299] Negotiation successful
0620 09:28:43.642902 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":76,"threads_started":1}
I20260620 09:28:43.643170 30787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:43.638389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.129:43333 (local address 127.0.0.1:43982)
0620 09:28:43.638712 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0620 09:28:43.638756 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 09:28:43.638823 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:28:43.639112 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:28:43.639116 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:28:43.639119 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:28:43.639193 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:28:43.639195 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:28:43.640548 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:28:43.640549 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:28:43.641658 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:28:43.641660 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:28:43.642674 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:28:43.642674 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:28:43.642744 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:28:43.642750 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 09:28:43.642798 (+ 48us) client_negotiation.cc:240] Negotiation successful
0620 09:28:43.643106 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"mutex_wait_us":245,"thread_start_us":96,"threads_started":1}
I20260620 09:28:43.643428 30792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:43.638697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57336 (local address 127.16.43.131:33529)
0620 09:28:43.638840 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 09:28:43.638844 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:28:43.639441 (+ 597us) server_negotiation.cc:407] Connection header received
0620 09:28:43.639499 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:28:43.639501 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:28:43.639518 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:28:43.639623 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:28:43.639723 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:28:43.640697 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:28:43.641825 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:28:43.642514 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:28:43.642576 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:28:43.642633 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 09:28:43.643282 (+ 649us) server_negotiation.cc:299] Negotiation successful
0620 09:28:43.643342 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":81,"threads_started":1}
I20260620 09:28:43.643994 30576 tablet_service.cc:1467] Processing CreateTablet for tablet d8d96d70e12445f1aa93fdb71d97100b (DEFAULT_TABLE table=test-table [id=add96c37041c41d8a37bf6f9ca1cb60b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:28:43.643982 30442 tablet_service.cc:1467] Processing CreateTablet for tablet d8d96d70e12445f1aa93fdb71d97100b (DEFAULT_TABLE table=test-table [id=add96c37041c41d8a37bf6f9ca1cb60b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:28:43.644135 30710 tablet_service.cc:1467] Processing CreateTablet for tablet d8d96d70e12445f1aa93fdb71d97100b (DEFAULT_TABLE table=test-table [id=add96c37041c41d8a37bf6f9ca1cb60b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:28:43.644349 30576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8d96d70e12445f1aa93fdb71d97100b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:28:43.644345 30442 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8d96d70e12445f1aa93fdb71d97100b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:28:43.644413 30710 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8d96d70e12445f1aa93fdb71d97100b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:28:43.646837 30793 tablet_bootstrap.cc:492] T d8d96d70e12445f1aa93fdb71d97100b P 52358e0cc7da4808a747d83dab337afd: Bootstrap starting.
I20260620 09:28:43.647519 30793 tablet_bootstrap.cc:654] T d8d96d70e12445f1aa93fdb71d97100b P 52358e0cc7da4808a747d83dab337afd: Neither blocks nor log segments found. Creating new log.
I20260620 09:28:43.647666 30794 tablet_bootstrap.cc:492] T d8d96d70e12445f1aa93fdb71d97100b P 55de9465487443d0bcd713075a4d8438: Bootstrap starting.
I20260620 09:28:43.647884 30795 tablet_bootstrap.cc:492] T d8d96d70e12445f1aa93fdb71d97100b P ca9f7e88ebe741259e36b11e1ddedca5: Bootstrap starting.
I20260620 09:28:43.648224 30793 log.cc:824] T d8d96d70e12445f1aa93fdb71d97100b P 52358e0cc7da4808a747d83dab337afd: Log is configured to *not* fsync() on all Append() calls
I20260620 09:28:43.648420 30794 tablet_bootstrap.cc:654] T d8d96d70e12445f1aa93fdb71d97100b P 55de9465487443d0bcd713075a4d8438: Neither blocks nor log segments found. Creating new log.
I20260620 09:28:43.648536 30795 tablet_bootstrap.cc:654] T d8d96d70e12445f1aa93fdb71d97100b P ca9f7e88ebe741259e36b11e1ddedca5: Neither blocks nor log segments found. Creating new log.
I20260620 09:28:43.649022 30794 log.cc:824] T d8d96d70e12445f1aa93fdb71d97100b P 55de9465487443d0bcd713075a4d8438: Log is configured to *not* fsync() on all Append() calls
I20260620 09:28:43.649106 30793 tablet_bootstrap.cc:492] T d8d96d70e12445f1aa93fdb71d97100b P 52358e0cc7da4808a747d83dab337afd: No bootstrap required, opened a new log
I20260620 09:28:43.649073 30795 log.cc:824] T d8d96d70e12445f1aa93fdb71d97100b P ca9f7e88ebe741259e36b11e1ddedca5: Log is configured to *not* fsync() on all Append() calls
I20260620 09:28:43.649164 30793 ts_tablet_manager.cc:1397] T d8d96d70e12445f1aa93fdb71d97100b P 52358e0cc7da4808a747d83dab337afd: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 09:28:43.649844 30795 tablet_bootstrap.cc:492] T d8d96d70e12445f1aa93fdb71d97100b P ca9f7e88ebe741259e36b11e1ddedca5: No bootstrap required, opened a new log
I20260620 09:28:43.649919 30795 ts_tablet_manager.cc:1397] T d8d96d70e12445f1aa93fdb71d97100b P ca9f7e88ebe741259e36b11e1ddedca5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 09:28:43.651183 30793 raft_consensus.cc:348] T d8d96d70e12445f1aa93fdb71d97100b P 52358e0cc7da4808a747d83dab337afd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55de9465487443d0bcd713075a4d8438" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 43333 } } peers { permanent_uuid: "52358e0cc7da4808a747d83dab337afd" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 38497 } } peers { permanent_uuid: "ca9f7e88ebe741259e36b11e1ddedca5" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 33529 } }
I20260620 09:28:43.651383 30793 raft_consensus.cc:374] T d8d96d70e12445f1aa93fdb71d97100b P 52358e0cc7da4808a747d83dab337afd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:28:43.651420 30793 raft_consensus.cc:729] T d8d96d70e12445f1aa93fdb71d97100b P 52358e0cc7da4808a747d83dab337afd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52358e0cc7da4808a747d83dab337afd, State: Initialized, Role: FOLLOWER
I20260620 09:28:43.651574 30793 consensus_queue.cc:260] T d8d96d70e12445f1aa93fdb71d97100b P 52358e0cc7da4808a747d83dab337afd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55de9465487443d0bcd713075a4d8438" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 43333 } } peers { permanent_uuid: "52358e0cc7da4808a747d83dab337afd" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 38497 } } peers { permanent_uuid: "ca9f7e88ebe741259e36b11e1ddedca5" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 33529 } }
I20260620 09:28:43.651870 30793 ts_tablet_manager.cc:1428] T d8d96d70e12445f1aa93fdb71d97100b P 52358e0cc7da4808a747d83dab337afd: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 09:28:43.651901 30794 tablet_bootstrap.cc:492] T d8d96d70e12445f1aa93fdb71d97100b P 55de9465487443d0bcd713075a4d8438: No bootstrap required, opened a new log
I20260620 09:28:43.651978 30794 ts_tablet_manager.cc:1397] T d8d96d70e12445f1aa93fdb71d97100b P 55de9465487443d0bcd713075a4d8438: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260620 09:28:43.652001 30642 heartbeater.cc:503] Master 127.16.43.190:38073 was elected leader, sending a full tablet report...
I20260620 09:28:43.651932 30795 raft_consensus.cc:348] T d8d96d70e12445f1aa93fdb71d97100b P ca9f7e88ebe741259e36b11e1ddedca5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55de9465487443d0bcd713075a4d8438" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 43333 } } peers { permanent_uuid: "52358e0cc7da4808a747d83dab337afd" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 38497 } } peers { permanent_uuid: "ca9f7e88ebe741259e36b11e1ddedca5" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 33529 } }
I20260620 09:28:43.652096 30795 raft_consensus.cc:374] T d8d96d70e12445f1aa93fdb71d97100b P ca9f7e88ebe741259e36b11e1ddedca5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:28:43.652137 30795 raft_consensus.cc:729] T d8d96d70e12445f1aa93fdb71d97100b P ca9f7e88ebe741259e36b11e1ddedca5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca9f7e88ebe741259e36b11e1ddedca5, State: Initialized, Role: FOLLOWER
I20260620 09:28:43.652244 30795 consensus_queue.cc:260] T d8d96d70e12445f1aa93fdb71d97100b P ca9f7e88ebe741259e36b11e1ddedca5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55de9465487443d0bcd713075a4d8438" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 43333 } } peers { permanent_uuid: "52358e0cc7da4808a747d83dab337afd" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 38497 } } peers { permanent_uuid: "ca9f7e88ebe741259e36b11e1ddedca5" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 33529 } }
I20260620 09:28:43.652544 30776 heartbeater.cc:503] Master 127.16.43.190:38073 was elected leader, sending a full tablet report...
I20260620 09:28:43.652807 30795 ts_tablet_manager.cc:1428] T d8d96d70e12445f1aa93fdb71d97100b P ca9f7e88ebe741259e36b11e1ddedca5: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 09:28:43.654017 30794 raft_consensus.cc:348] T d8d96d70e12445f1aa93fdb71d97100b P 55de9465487443d0bcd713075a4d8438 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55de9465487443d0bcd713075a4d8438" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 43333 } } peers { permanent_uuid: "52358e0cc7da4808a747d83dab337afd" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 38497 } } peers { permanent_uuid: "ca9f7e88ebe741259e36b11e1ddedca5" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 33529 } }
I20260620 09:28:43.654142 30794 raft_consensus.cc:374] T d8d96d70e12445f1aa93fdb71d97100b P 55de9465487443d0bcd713075a4d8438 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:28:43.654165 30794 raft_consensus.cc:729] T d8d96d70e12445f1aa93fdb71d97100b P 55de9465487443d0bcd713075a4d8438 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55de9465487443d0bcd713075a4d8438, State: Initialized, Role: FOLLOWER
I20260620 09:28:43.654292 30794 consensus_queue.cc:260] T d8d96d70e12445f1aa93fdb71d97100b P 55de9465487443d0bcd713075a4d8438 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55de9465487443d0bcd713075a4d8438" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 43333 } } peers { permanent_uuid: "52358e0cc7da4808a747d83dab337afd" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 38497 } } peers { permanent_uuid: "ca9f7e88ebe741259e36b11e1ddedca5" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 33529 } }
I20260620 09:28:43.654588 30794 ts_tablet_manager.cc:1428] T d8d96d70e12445f1aa93fdb71d97100b P 55de9465487443d0bcd713075a4d8438: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 09:28:43.654609 30508 heartbeater.cc:503] Master 127.16.43.190:38073 was elected leader, sending a full tablet report...
W20260620 09:28:43.697243 30643 tablet.cc:2369] T d8d96d70e12445f1aa93fdb71d97100b P 52358e0cc7da4808a747d83dab337afd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 09:28:43.758010 30509 tablet.cc:2369] T d8d96d70e12445f1aa93fdb71d97100b P 55de9465487443d0bcd713075a4d8438: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 09:28:43.785115 30799 raft_consensus.cc:482] T d8d96d70e12445f1aa93fdb71d97100b P 52358e0cc7da4808a747d83dab337afd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:28:43.785233 30799 raft_consensus.cc:504] T d8d96d70e12445f1aa93fdb71d97100b P 52358e0cc7da4808a747d83dab337afd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55de9465487443d0bcd713075a4d8438" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 43333 } } peers { permanent_uuid: "52358e0cc7da4808a747d83dab337afd" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 38497 } } peers { permanent_uuid: "ca9f7e88ebe741259e36b11e1ddedca5" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 33529 } }
I20260620 09:28:43.785655 30799 leader_election.cc:288] T d8d96d70e12445f1aa93fdb71d97100b P 52358e0cc7da4808a747d83dab337afd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 55de9465487443d0bcd713075a4d8438 (127.16.43.129:43333), ca9f7e88ebe741259e36b11e1ddedca5 (127.16.43.131:33529)
I20260620 09:28:43.789570 30792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:43.785921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.130:44441 (local address 127.16.43.131:33529)
0620 09:28:43.785971 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 09:28:43.785973 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:28:43.786211 (+ 238us) server_negotiation.cc:407] Connection header received
0620 09:28:43.786300 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:28:43.786302 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:28:43.786311 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:28:43.786379 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:28:43.786548 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:28:43.787251 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:28:43.788654 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:28:43.789226 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:28:43.789292 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:28:43.789300 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 09:28:43.789462 (+ 162us) server_negotiation.cc:299] Negotiation successful
0620 09:28:43.789498 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 09:28:43.789570 30789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:43.785920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.130:44283 (local address 127.16.43.129:43333)
0620 09:28:43.785966 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 09:28:43.785968 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:28:43.786007 (+ 39us) server_negotiation.cc:407] Connection header received
0620 09:28:43.786104 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:28:43.786105 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:28:43.786115 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:28:43.786181 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:28:43.786350 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:28:43.787183 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:28:43.788654 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:28:43.789207 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:28:43.789291 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:28:43.789300 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 09:28:43.789458 (+ 158us) server_negotiation.cc:299] Negotiation successful
0620 09:28:43.789496 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 09:28:43.789589 30644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:43.785862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.129:43333 (local address 127.16.43.130:44283)
0620 09:28:43.785929 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0620 09:28:43.785944 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 09:28:43.786026 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:28:43.786196 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:28:43.786197 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:28:43.786199 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:28:43.786275 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:28:43.786277 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:28:43.787264 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:28:43.787265 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:28:43.788547 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:28:43.788550 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:28:43.789299 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:28:43.789300 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:28:43.789383 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:28:43.789385 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 09:28:43.789442 (+ 57us) client_negotiation.cc:240] Negotiation successful
0620 09:28:43.789491 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":16}
I20260620 09:28:43.789597 30802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:43.785862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.131:33529 (local address 127.16.43.130:44441)
0620 09:28:43.786140 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0620 09:28:43.786148 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 09:28:43.786226 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:28:43.786384 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:28:43.786384 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:28:43.786386 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:28:43.786460 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:28:43.786462 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:28:43.787319 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:28:43.787320 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:28:43.788547 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:28:43.788550 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:28:43.789299 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:28:43.789300 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:28:43.789379 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:28:43.789384 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 09:28:43.789442 (+ 58us) client_negotiation.cc:240] Negotiation successful
0620 09:28:43.789490 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"mutex_wait_us":87,"thread_start_us":137,"threads_started":1}
I20260620 09:28:43.790053 30730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8d96d70e12445f1aa93fdb71d97100b" candidate_uuid: "52358e0cc7da4808a747d83dab337afd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca9f7e88ebe741259e36b11e1ddedca5" is_pre_election: true
I20260620 09:28:43.790158 30462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8d96d70e12445f1aa93fdb71d97100b" candidate_uuid: "52358e0cc7da4808a747d83dab337afd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55de9465487443d0bcd713075a4d8438" is_pre_election: true
I20260620 09:28:43.790239 30730 raft_consensus.cc:2454] T d8d96d70e12445f1aa93fdb71d97100b P ca9f7e88ebe741259e36b11e1ddedca5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52358e0cc7da4808a747d83dab337afd in term 0.
I20260620 09:28:43.790285 30462 raft_consensus.cc:2454] T d8d96d70e12445f1aa93fdb71d97100b P 55de9465487443d0bcd713075a4d8438 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52358e0cc7da4808a747d83dab337afd in term 0.
I20260620 09:28:43.790477 30532 leader_election.cc:302] T d8d96d70e12445f1aa93fdb71d97100b P 52358e0cc7da4808a747d83dab337afd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52358e0cc7da4808a747d83dab337afd, ca9f7e88ebe741259e36b11e1ddedca5; no voters:
I20260620 09:28:43.790602 30799 raft_consensus.cc:2781] T d8d96d70e12445f1aa93fdb71d97100b P 52358e0cc7da4808a747d83dab337afd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 09:28:43.790657 30799 raft_consensus.cc:482] T d8d96d70e12445f1aa93fdb71d97100b P 52358e0cc7da4808a747d83dab337afd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 09:28:43.790688 30799 raft_consensus.cc:3037] T d8d96d70e12445f1aa93fdb71d97100b P 52358e0cc7da4808a747d83dab337afd [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:28:43.791424 30799 raft_consensus.cc:504] T d8d96d70e12445f1aa93fdb71d97100b P 52358e0cc7da4808a747d83dab337afd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55de9465487443d0bcd713075a4d8438" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 43333 } } peers { permanent_uuid: "52358e0cc7da4808a747d83dab337afd" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 38497 } } peers { permanent_uuid: "ca9f7e88ebe741259e36b11e1ddedca5" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 33529 } }
I20260620 09:28:43.791604 30799 leader_election.cc:288] T d8d96d70e12445f1aa93fdb71d97100b P 52358e0cc7da4808a747d83dab337afd [CANDIDATE]: Term 1 election: Requested vote from peers 55de9465487443d0bcd713075a4d8438 (127.16.43.129:43333), ca9f7e88ebe741259e36b11e1ddedca5 (127.16.43.131:33529)
I20260620 09:28:43.791759 30730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8d96d70e12445f1aa93fdb71d97100b" candidate_uuid: "52358e0cc7da4808a747d83dab337afd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca9f7e88ebe741259e36b11e1ddedca5"
I20260620 09:28:43.791775 30462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8d96d70e12445f1aa93fdb71d97100b" candidate_uuid: "52358e0cc7da4808a747d83dab337afd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55de9465487443d0bcd713075a4d8438"
I20260620 09:28:43.791847 30462 raft_consensus.cc:3037] T d8d96d70e12445f1aa93fdb71d97100b P 55de9465487443d0bcd713075a4d8438 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:28:43.791847 30730 raft_consensus.cc:3037] T d8d96d70e12445f1aa93fdb71d97100b P ca9f7e88ebe741259e36b11e1ddedca5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:28:43.792726 30730 raft_consensus.cc:2454] T d8d96d70e12445f1aa93fdb71d97100b P ca9f7e88ebe741259e36b11e1ddedca5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52358e0cc7da4808a747d83dab337afd in term 1.
I20260620 09:28:43.792726 30462 raft_consensus.cc:2454] T d8d96d70e12445f1aa93fdb71d97100b P 55de9465487443d0bcd713075a4d8438 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52358e0cc7da4808a747d83dab337afd in term 1.
I20260620 09:28:43.792938 30532 leader_election.cc:302] T d8d96d70e12445f1aa93fdb71d97100b P 52358e0cc7da4808a747d83dab337afd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52358e0cc7da4808a747d83dab337afd, ca9f7e88ebe741259e36b11e1ddedca5; no voters:
I20260620 09:28:43.793026 30799 raft_consensus.cc:2781] T d8d96d70e12445f1aa93fdb71d97100b P 52358e0cc7da4808a747d83dab337afd [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:28:43.793187 30799 raft_consensus.cc:686] T d8d96d70e12445f1aa93fdb71d97100b P 52358e0cc7da4808a747d83dab337afd [term 1 LEADER]: Becoming Leader. State: Replica: 52358e0cc7da4808a747d83dab337afd, State: Running, Role: LEADER
I20260620 09:28:43.793295 30799 consensus_queue.cc:237] T d8d96d70e12445f1aa93fdb71d97100b P 52358e0cc7da4808a747d83dab337afd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55de9465487443d0bcd713075a4d8438" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 43333 } } peers { permanent_uuid: "52358e0cc7da4808a747d83dab337afd" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 38497 } } peers { permanent_uuid: "ca9f7e88ebe741259e36b11e1ddedca5" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 33529 } }
I20260620 09:28:43.794144 30317 catalog_manager.cc:5495] T d8d96d70e12445f1aa93fdb71d97100b P 52358e0cc7da4808a747d83dab337afd reported cstate change: term changed from 0 to 1, leader changed from <none> to 52358e0cc7da4808a747d83dab337afd (127.16.43.130). New cstate: current_term: 1 leader_uuid: "52358e0cc7da4808a747d83dab337afd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55de9465487443d0bcd713075a4d8438" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 43333 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52358e0cc7da4808a747d83dab337afd" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 38497 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca9f7e88ebe741259e36b11e1ddedca5" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 33529 } health_report { overall_health: UNKNOWN } } }
I20260620 09:28:43.827812 30790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:43.825189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49082 (local address 127.16.43.130:38497)
0620 09:28:43.825265 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0620 09:28:43.825267 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:28:43.825275 (+ 8us) server_negotiation.cc:407] Connection header received
0620 09:28:43.825314 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:28:43.825315 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:28:43.825333 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:28:43.825403 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:28:43.825629 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:28:43.826284 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:28:43.827072 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:28:43.827324 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:28:43.827376 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:28:43.827490 (+ 114us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:28:43.827634 (+ 144us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:28:43.827701 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 09:28:43.827713 (+ 12us) server_negotiation.cc:299] Negotiation successful
0620 09:28:43.827731 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 09:28:43.829691 30730 raft_consensus.cc:1261] T d8d96d70e12445f1aa93fdb71d97100b P ca9f7e88ebe741259e36b11e1ddedca5 [term 1 FOLLOWER]: Refusing update from remote peer 52358e0cc7da4808a747d83dab337afd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:28:43.829691 30462 raft_consensus.cc:1261] T d8d96d70e12445f1aa93fdb71d97100b P 55de9465487443d0bcd713075a4d8438 [term 1 FOLLOWER]: Refusing update from remote peer 52358e0cc7da4808a747d83dab337afd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:28:43.830106 30799 consensus_queue.cc:1035] T d8d96d70e12445f1aa93fdb71d97100b P 52358e0cc7da4808a747d83dab337afd [LEADER]: Connected to new peer: Peer: permanent_uuid: "55de9465487443d0bcd713075a4d8438" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 43333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:28:43.830255 30803 consensus_queue.cc:1035] T d8d96d70e12445f1aa93fdb71d97100b P 52358e0cc7da4808a747d83dab337afd [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca9f7e88ebe741259e36b11e1ddedca5" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 33529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:28:43.832679 30806 mvcc.cc:205] Tried to move back new op lower bound from 7298857876803268608 to 7298857876657913856. Current Snapshot: MvccSnapshot[applied={T|T < 7298857876803268608}]
I20260620 09:28:43.832778 30808 mvcc.cc:205] Tried to move back new op lower bound from 7298857876803268608 to 7298857876657913856. Current Snapshot: MvccSnapshot[applied={T|T < 7298857876803268608}]
I20260620 09:28:43.832989 30807 mvcc.cc:205] Tried to move back new op lower bound from 7298857876803268608 to 7298857876657913856. Current Snapshot: MvccSnapshot[applied={T|T < 7298857876803268608}]
I20260620 09:28:43.835403 30316 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34462:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:28:43.835511 30316 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34462:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:28:43.836278 30316 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 516727c9072f4e0f8034238b71a751d3: Sending DeleteTablet for 3 replicas of tablet d8d96d70e12445f1aa93fdb71d97100b
I20260620 09:28:43.836611 30442 tablet_service.cc:1514] Processing DeleteTablet for tablet d8d96d70e12445f1aa93fdb71d97100b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:28:43 UTC) from {username='kudu', principal='kudu/127.16.43.190@KRBTEST.COM'} at 127.0.0.1:43982
I20260620 09:28:43.836661 30710 tablet_service.cc:1514] Processing DeleteTablet for tablet d8d96d70e12445f1aa93fdb71d97100b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:28:43 UTC) from {username='kudu', principal='kudu/127.16.43.190@KRBTEST.COM'} at 127.0.0.1:57336
I20260620 09:28:43.836719 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 30379
I20260620 09:28:43.837244 30576 tablet_service.cc:1514] Processing DeleteTablet for tablet d8d96d70e12445f1aa93fdb71d97100b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:28:43 UTC) from {username='kudu', principal='kudu/127.16.43.190@KRBTEST.COM'} at 127.0.0.1:49074
I20260620 09:28:43.837373 30817 tablet_replica.cc:330] T d8d96d70e12445f1aa93fdb71d97100b P ca9f7e88ebe741259e36b11e1ddedca5: stopping tablet replica
I20260620 09:28:43.837476 30817 raft_consensus.cc:2229] T d8d96d70e12445f1aa93fdb71d97100b P ca9f7e88ebe741259e36b11e1ddedca5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:28:43.837620 30817 raft_consensus.cc:2258] T d8d96d70e12445f1aa93fdb71d97100b P ca9f7e88ebe741259e36b11e1ddedca5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:28:43.838143 30817 ts_tablet_manager.cc:1905] T d8d96d70e12445f1aa93fdb71d97100b P ca9f7e88ebe741259e36b11e1ddedca5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:28:43.840066 30817 ts_tablet_manager.cc:1918] T d8d96d70e12445f1aa93fdb71d97100b P ca9f7e88ebe741259e36b11e1ddedca5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:28:43.840159 30817 log.cc:1196] T d8d96d70e12445f1aa93fdb71d97100b P ca9f7e88ebe741259e36b11e1ddedca5: Deleting WAL directory at /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-2/wal/wals/d8d96d70e12445f1aa93fdb71d97100b
I20260620 09:28:43.840498 30817 ts_tablet_manager.cc:1939] T d8d96d70e12445f1aa93fdb71d97100b P ca9f7e88ebe741259e36b11e1ddedca5: Deleting consensus metadata
I20260620 09:28:43.840983 30304 catalog_manager.cc:4841] TS ca9f7e88ebe741259e36b11e1ddedca5 (127.16.43.131:33529): tablet d8d96d70e12445f1aa93fdb71d97100b (table test-table [id=add96c37041c41d8a37bf6f9ca1cb60b]) successfully deleted
W20260620 09:28:43.842442 30301 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.43.129:43333 (error 108)
I20260620 09:28:43.842971 30788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:43.842831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.129:43333 (local address 127.0.0.1:43986)
0620 09:28:43.842891 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0620 09:28:43.842938 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.129:43333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260620 09:28:43.843016 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 30513
W20260620 09:28:43.843109 30301 catalog_manager.cc:4584] TS 55de9465487443d0bcd713075a4d8438 (127.16.43.129:43333): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d8d96d70e12445f1aa93fdb71d97100b: Network error: Client connection negotiation failed: client connection to 127.16.43.129:43333: connect: Connection refused (error 111)
I20260620 09:28:43.849298 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 30647
I20260620 09:28:43.849337 30788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:43.849213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.130:38497 (local address 127.0.0.1:49086)
0620 09:28:43.849268 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0620 09:28:43.849303 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.130:38497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260620 09:28:43.853931 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 30285
I20260620 09:28:43.859115 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.43.190:38073
--webserver_interface=127.16.43.190
--webserver_port=43503
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.190.keytab
--principal=kudu/127.16.43.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.43.148:42633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.43.190:38073
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:28:43.940827 30820 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:28:43.941044 30820 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:28:43.941069 30820 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:28:43.942257 30820 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:28:43.942305 30820 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:28:43.942319 30820 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:28:43.942332 30820 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:28:43.942344 30820 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:28:43.943871 30820 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:42633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.16.43.190:38073
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.190.keytab
--principal=kudu/127.16.43.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.43.190:38073
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.43.190
--webserver_port=43503
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30820
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:28:43.944145 30820 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:28:43.944387 30820 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:28:43.953313 30827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:28:43.953313 30829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:28:43.953313 30826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:28:43.953348 30820 server_base.cc:1034] running on GCE node
I20260620 09:28:43.953902 30820 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:28:43.954190 30820 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:28:43.955323 30820 hybrid_clock.cc:648] HybridClock initialized: now 1781947723955303 us; error 33 us; skew 500 ppm
Jun 20 09:28:43 dist-test-slave-706q krb5kdc[30272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947723, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:28:43.957912 30820 init.cc:377] Logged in from keytab as kudu/127.16.43.190@KRBTEST.COM (short username kudu)
I20260620 09:28:43.958386 30820 webserver.cc:460] Webserver started at http://127.16.43.190:43503/ using document root <none> and password file <none>
I20260620 09:28:43.958645 30820 fs_manager.cc:359] Metadata directory not provided
I20260620 09:28:43.958731 30820 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:28:43.960217 30820 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 09:28:43.960831 30836 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:28:43.960995 30820 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 09:28:43.961097 30820 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/master-0/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/master-0/wal
uuid: "516727c9072f4e0f8034238b71a751d3"
format_stamp: "Formatted at 2026-06-20 09:28:43 on dist-test-slave-706q"
server_key: "1bf6a9adabdb7e214fbfe067da2bb3f6"
server_key_iv: "bc521b0d457322537acc73bfbfeec5ae"
server_key_version: "encryptionkey@0"
I20260620 09:28:43.961503 30820 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:28:43.983512 30820 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:28:43.985468 30820 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:28:43.985631 30820 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:28:43.989543 30820 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.190:38073
I20260620 09:28:43.989581 30888 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.190:38073 every 8 connection(s)
I20260620 09:28:43.989985 30820 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
I20260620 09:28:43.991659 30889 sys_catalog.cc:263] Verifying existing consensus state
I20260620 09:28:43.992245 30889 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 516727c9072f4e0f8034238b71a751d3: Bootstrap starting.
I20260620 09:28:43.994385 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 30820
I20260620 09:28:43.994668 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.43.129:43333
--local_ip_for_outbound_sockets=127.16.43.129
--tserver_master_addrs=127.16.43.190:38073
--webserver_port=42289
--webserver_interface=127.16.43.129
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.129.keytab
--principal=kudu/127.16.43.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.43.148:42633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 09:28:44.000952 30889 log.cc:824] T 00000000000000000000000000000000 P 516727c9072f4e0f8034238b71a751d3: Log is configured to *not* fsync() on all Append() calls
I20260620 09:28:44.002926 30889 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 516727c9072f4e0f8034238b71a751d3: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 09:28:44.003253 30889 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 516727c9072f4e0f8034238b71a751d3: Bootstrap complete.
I20260620 09:28:44.005584 30889 raft_consensus.cc:348] T 00000000000000000000000000000000 P 516727c9072f4e0f8034238b71a751d3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "516727c9072f4e0f8034238b71a751d3" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 38073 } }
I20260620 09:28:44.005831 30889 raft_consensus.cc:729] T 00000000000000000000000000000000 P 516727c9072f4e0f8034238b71a751d3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 516727c9072f4e0f8034238b71a751d3, State: Initialized, Role: FOLLOWER
I20260620 09:28:44.006011 30889 consensus_queue.cc:260] T 00000000000000000000000000000000 P 516727c9072f4e0f8034238b71a751d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "516727c9072f4e0f8034238b71a751d3" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 38073 } }
I20260620 09:28:44.006073 30889 raft_consensus.cc:388] T 00000000000000000000000000000000 P 516727c9072f4e0f8034238b71a751d3 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:28:44.006127 30889 raft_consensus.cc:482] T 00000000000000000000000000000000 P 516727c9072f4e0f8034238b71a751d3 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:28:44.006177 30889 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 516727c9072f4e0f8034238b71a751d3 [term 1 FOLLOWER]: Advancing to term 2
I20260620 09:28:44.007247 30889 raft_consensus.cc:504] T 00000000000000000000000000000000 P 516727c9072f4e0f8034238b71a751d3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "516727c9072f4e0f8034238b71a751d3" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 38073 } }
I20260620 09:28:44.007364 30889 leader_election.cc:302] T 00000000000000000000000000000000 P 516727c9072f4e0f8034238b71a751d3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 516727c9072f4e0f8034238b71a751d3; no voters:
I20260620 09:28:44.007613 30889 leader_election.cc:288] T 00000000000000000000000000000000 P 516727c9072f4e0f8034238b71a751d3 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 09:28:44.007638 30893 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 516727c9072f4e0f8034238b71a751d3 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 09:28:44.007753 30893 raft_consensus.cc:686] T 00000000000000000000000000000000 P 516727c9072f4e0f8034238b71a751d3 [term 2 LEADER]: Becoming Leader. State: Replica: 516727c9072f4e0f8034238b71a751d3, State: Running, Role: LEADER
I20260620 09:28:44.007891 30889 sys_catalog.cc:564] T 00000000000000000000000000000000 P 516727c9072f4e0f8034238b71a751d3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:28:44.007915 30893 consensus_queue.cc:237] T 00000000000000000000000000000000 P 516727c9072f4e0f8034238b71a751d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "516727c9072f4e0f8034238b71a751d3" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 38073 } }
I20260620 09:28:44.008435 30894 sys_catalog.cc:455] T 00000000000000000000000000000000 P 516727c9072f4e0f8034238b71a751d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "516727c9072f4e0f8034238b71a751d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "516727c9072f4e0f8034238b71a751d3" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 38073 } } }
I20260620 09:28:44.008525 30894 sys_catalog.cc:458] T 00000000000000000000000000000000 P 516727c9072f4e0f8034238b71a751d3 [sys.catalog]: This master's current role is: LEADER
I20260620 09:28:44.008725 30902 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:28:44.008442 30895 sys_catalog.cc:455] T 00000000000000000000000000000000 P 516727c9072f4e0f8034238b71a751d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 516727c9072f4e0f8034238b71a751d3. Latest consensus state: current_term: 2 leader_uuid: "516727c9072f4e0f8034238b71a751d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "516727c9072f4e0f8034238b71a751d3" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 38073 } } }
I20260620 09:28:44.008857 30895 sys_catalog.cc:458] T 00000000000000000000000000000000 P 516727c9072f4e0f8034238b71a751d3 [sys.catalog]: This master's current role is: LEADER
I20260620 09:28:44.009639 30902 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:28:44.009755 30902 catalog_manager.cc:1259] Loaded cluster ID: 8b0481e1fb5546209dc536267159980d
I20260620 09:28:44.009846 30902 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:28:44.010411 30902 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260620 09:28:44.015592 30902 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260620 09:28:44.018186 30902 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:28:44.018534 30902 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 516727c9072f4e0f8034238b71a751d3: Loaded TSK: 0
I20260620 09:28:44.018819 30902 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 09:28:44.081060 30891 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:28:44.081295 30891 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:28:44.081331 30891 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:28:44.082508 30891 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:28:44.082574 30891 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:28:44.082623 30891 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.43.129
I20260620 09:28:44.084290 30891 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:42633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.129.keytab
--principal=kudu/127.16.43.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.43.129:43333
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.43.129
--webserver_port=42289
--tserver_master_addrs=127.16.43.190:38073
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30891
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.43.129
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:28:44.084570 30891 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:28:44.084805 30891 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:28:44.089998 30919 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:28:44.089998 30917 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:28:44.090035 30916 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:28:44.090534 30891 server_base.cc:1034] running on GCE node
I20260620 09:28:44.090745 30891 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:28:44.091008 30891 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:28:44.092190 30891 hybrid_clock.cc:648] HybridClock initialized: now 1781947724092168 us; error 36 us; skew 500 ppm
Jun 20 09:28:44 dist-test-slave-706q krb5kdc[30272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947724, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:28:44.094903 30891 init.cc:377] Logged in from keytab as kudu/127.16.43.129@KRBTEST.COM (short username kudu)
I20260620 09:28:44.095355 30891 webserver.cc:460] Webserver started at http://127.16.43.129:42289/ using document root <none> and password file <none>
I20260620 09:28:44.095631 30891 fs_manager.cc:359] Metadata directory not provided
I20260620 09:28:44.095711 30891 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:28:44.097223 30891 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:28:44.097963 30926 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:28:44.098152 30891 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:28:44.098263 30891 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-0/wal
uuid: "55de9465487443d0bcd713075a4d8438"
format_stamp: "Formatted at 2026-06-20 09:28:43 on dist-test-slave-706q"
server_key: "b8cf4182d8252833daa52bc821848630"
server_key_iv: "1d181cee9ea91d529ae9d23154d68401"
server_key_version: "encryptionkey@0"
I20260620 09:28:44.098639 30891 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:28:44.110637 30891 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:28:44.112942 30891 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:28:44.113129 30891 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:28:44.113377 30891 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:28:44.113793 30933 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 09:28:44.115737 30891 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 09:28:44.115798 30891 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260620 09:28:44.115857 30891 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 09:28:44.116505 30891 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 09:28:44.116559 30891 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260620 09:28:44.116650 30933 tablet_bootstrap.cc:492] T d8d96d70e12445f1aa93fdb71d97100b P 55de9465487443d0bcd713075a4d8438: Bootstrap starting.
I20260620 09:28:44.123711 30891 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.129:43333
I20260620 09:28:44.123842 31040 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.129:43333 every 8 connection(s)
I20260620 09:28:44.124188 30891 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-0/data/info.pb
I20260620 09:28:44.128348 30933 log.cc:824] T d8d96d70e12445f1aa93fdb71d97100b P 55de9465487443d0bcd713075a4d8438: Log is configured to *not* fsync() on all Append() calls
I20260620 09:28:44.130148 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 30891
I20260620 09:28:44.130427 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.43.130:38497
--local_ip_for_outbound_sockets=127.16.43.130
--tserver_master_addrs=127.16.43.190:38073
--webserver_port=36893
--webserver_interface=127.16.43.130
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.130.keytab
--principal=kudu/127.16.43.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.43.148:42633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 09:28:44.131031 30933 tablet_bootstrap.cc:492] T d8d96d70e12445f1aa93fdb71d97100b P 55de9465487443d0bcd713075a4d8438: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 09:28:44.131335 30933 tablet_bootstrap.cc:492] T d8d96d70e12445f1aa93fdb71d97100b P 55de9465487443d0bcd713075a4d8438: Bootstrap complete.
I20260620 09:28:44.132599 30933 ts_tablet_manager.cc:1397] T d8d96d70e12445f1aa93fdb71d97100b P 55de9465487443d0bcd713075a4d8438: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.013s
I20260620 09:28:44.134076 30933 raft_consensus.cc:348] T d8d96d70e12445f1aa93fdb71d97100b P 55de9465487443d0bcd713075a4d8438 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55de9465487443d0bcd713075a4d8438" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 43333 } } peers { permanent_uuid: "52358e0cc7da4808a747d83dab337afd" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 38497 } } peers { permanent_uuid: "ca9f7e88ebe741259e36b11e1ddedca5" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 33529 } }
I20260620 09:28:44.134850 30933 raft_consensus.cc:729] T d8d96d70e12445f1aa93fdb71d97100b P 55de9465487443d0bcd713075a4d8438 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55de9465487443d0bcd713075a4d8438, State: Initialized, Role: FOLLOWER
I20260620 09:28:44.135129 30933 consensus_queue.cc:260] T d8d96d70e12445f1aa93fdb71d97100b P 55de9465487443d0bcd713075a4d8438 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55de9465487443d0bcd713075a4d8438" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 43333 } } peers { permanent_uuid: "52358e0cc7da4808a747d83dab337afd" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 38497 } } peers { permanent_uuid: "ca9f7e88ebe741259e36b11e1ddedca5" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 33529 } }
I20260620 09:28:44.135492 30933 ts_tablet_manager.cc:1428] T d8d96d70e12445f1aa93fdb71d97100b P 55de9465487443d0bcd713075a4d8438: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
Jun 20 09:28:44 dist-test-slave-706q krb5kdc[30272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947724, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.129@KRBTEST.COM for kudu/127.16.43.190@KRBTEST.COM
I20260620 09:28:44.141312 31044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:44.125039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.129:47549 (local address 127.16.43.190:38073)
0620 09:28:44.127784 (+ 2745us) server_negotiation.cc:206] Beginning negotiation
0620 09:28:44.127791 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 09:28:44.127824 (+ 33us) server_negotiation.cc:407] Connection header received
0620 09:28:44.127939 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:28:44.127943 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:28:44.127986 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:28:44.128149 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:28:44.128977 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:28:44.129911 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:28:44.133185 (+ 3274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:28:44.133869 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:28:44.133954 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:28:44.136837 (+ 2883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:28:44.136842 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:28:44.136843 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:28:44.136858 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:28:44.139165 (+ 2307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:28:44.139752 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:28:44.139756 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:28:44.139756 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:28:44.139871 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:28:44.140205 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:28:44.140206 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:28:44.140207 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:28:44.140503 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:28:44.140583 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 09:28:44.140963 (+ 380us) server_negotiation.cc:299] Negotiation successful
0620 09:28:44.141103 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2682,"thread_start_us":101,"threads_started":1}
I20260620 09:28:44.141366 31043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:44.124935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:38073 (local address 127.16.43.129:47549)
0620 09:28:44.125139 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0620 09:28:44.125191 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 09:28:44.125705 (+ 514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:28:44.128172 (+ 2467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:28:44.128176 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:28:44.128520 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:28:44.128840 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:28:44.128848 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:28:44.131637 (+ 2789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:28:44.131638 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:28:44.133060 (+ 1422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:28:44.133064 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:28:44.134077 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:28:44.134078 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:28:44.134246 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:28:44.134902 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:28:44.134914 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:28:44.136705 (+ 1791us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:28:44.139264 (+ 2559us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:28:44.139272 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:28:44.139275 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:28:44.139658 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:28:44.139956 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:28:44.139957 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:28:44.139958 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:28:44.140124 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:28:44.140578 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:28:44.140586 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:28:44.140767 (+ 181us) client_negotiation.cc:769] Sending connection context
0620 09:28:44.140891 (+ 124us) client_negotiation.cc:240] Negotiation successful
0620 09:28:44.141099 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":97,"threads_started":1}
I20260620 09:28:44.142877 31041 heartbeater.cc:347] Connected to a master server at 127.16.43.190:38073
I20260620 09:28:44.142982 31041 heartbeater.cc:464] Registering TS with master...
I20260620 09:28:44.143185 31041 heartbeater.cc:511] Master 127.16.43.190:38073 requested a full tablet report, sending...
I20260620 09:28:44.143981 30853 ts_manager.cc:194] Registered new tserver with Master: 55de9465487443d0bcd713075a4d8438 (127.16.43.129:43333)
I20260620 09:28:44.145213 30853 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.43.129@KRBTEST.COM'} at 127.16.43.129:47549
I20260620 09:28:44.146216 31041 heartbeater.cc:503] Master 127.16.43.190:38073 was elected leader, sending a full tablet report...
I20260620 09:28:44.155081 31048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:44.144611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.129:43333 (local address 127.0.0.1:43996)
0620 09:28:44.145074 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0620 09:28:44.145119 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 09:28:44.145184 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:28:44.151872 (+ 6688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:28:44.151875 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:28:44.151879 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:28:44.151945 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:28:44.151947 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:28:44.153161 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:28:44.153162 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:28:44.153973 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:28:44.153975 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:28:44.154826 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:28:44.154827 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:28:44.154893 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:28:44.154905 (+ 12us) client_negotiation.cc:769] Sending connection context
0620 09:28:44.154964 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 09:28:44.155011 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":98,"threads_started":1}
I20260620 09:28:44.155118 31049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:44.144893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43996 (local address 127.16.43.129:43333)
0620 09:28:44.151669 (+ 6776us) server_negotiation.cc:206] Beginning negotiation
0620 09:28:44.151674 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:28:44.151687 (+ 13us) server_negotiation.cc:407] Connection header received
0620 09:28:44.151728 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:28:44.151731 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:28:44.151750 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:28:44.151846 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:28:44.152025 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:28:44.153067 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:28:44.154057 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:28:44.154740 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:28:44.154814 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:28:44.154872 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 09:28:44.154993 (+ 121us) server_negotiation.cc:299] Negotiation successful
0620 09:28:44.155035 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6734,"thread_start_us":81,"threads_started":1}
I20260620 09:28:44.155579 30975 tablet_service.cc:1514] Processing DeleteTablet for tablet d8d96d70e12445f1aa93fdb71d97100b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:28:43 UTC) from {username='kudu', principal='kudu/127.16.43.190@KRBTEST.COM'} at 127.0.0.1:43996
I20260620 09:28:44.156133 31050 tablet_replica.cc:330] stopping tablet replica
I20260620 09:28:44.156243 31050 raft_consensus.cc:2229] T d8d96d70e12445f1aa93fdb71d97100b P 55de9465487443d0bcd713075a4d8438 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:28:44.156334 31050 raft_consensus.cc:2258] T d8d96d70e12445f1aa93fdb71d97100b P 55de9465487443d0bcd713075a4d8438 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:28:44.156652 31050 ts_tablet_manager.cc:1905] T d8d96d70e12445f1aa93fdb71d97100b P 55de9465487443d0bcd713075a4d8438: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:28:44.158571 31050 ts_tablet_manager.cc:1918] T d8d96d70e12445f1aa93fdb71d97100b P 55de9465487443d0bcd713075a4d8438: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:28:44.158679 31050 log.cc:1196] T d8d96d70e12445f1aa93fdb71d97100b P 55de9465487443d0bcd713075a4d8438: Deleting WAL directory at /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-0/wal/wals/d8d96d70e12445f1aa93fdb71d97100b
I20260620 09:28:44.158926 31050 ts_tablet_manager.cc:1939] T d8d96d70e12445f1aa93fdb71d97100b P 55de9465487443d0bcd713075a4d8438: Deleting consensus metadata
I20260620 09:28:44.159417 30837 catalog_manager.cc:4841] TS 55de9465487443d0bcd713075a4d8438 (127.16.43.129:43333): tablet d8d96d70e12445f1aa93fdb71d97100b (table test-table [id=add96c37041c41d8a37bf6f9ca1cb60b]) successfully deleted
W20260620 09:28:44.217855 31046 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:28:44.218109 31046 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:28:44.218149 31046 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:28:44.219323 31046 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:28:44.219394 31046 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:28:44.219440 31046 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.43.130
I20260620 09:28:44.221019 31046 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:42633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.130.keytab
--principal=kudu/127.16.43.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.43.130:38497
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.43.130
--webserver_port=36893
--tserver_master_addrs=127.16.43.190:38073
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31046
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.43.130
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:28:44.221295 31046 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:28:44.221540 31046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:28:44.226688 31055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:28:44.226677 31058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:28:44.226677 31056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:28:44.226841 31046 server_base.cc:1034] running on GCE node
I20260620 09:28:44.227144 31046 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:28:44.227434 31046 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:28:44.228616 31046 hybrid_clock.cc:648] HybridClock initialized: now 1781947724228609 us; error 34 us; skew 500 ppm
Jun 20 09:28:44 dist-test-slave-706q krb5kdc[30272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947724, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:28:44.231034 31046 init.cc:377] Logged in from keytab as kudu/127.16.43.130@KRBTEST.COM (short username kudu)
I20260620 09:28:44.231455 31046 webserver.cc:460] Webserver started at http://127.16.43.130:36893/ using document root <none> and password file <none>
I20260620 09:28:44.231700 31046 fs_manager.cc:359] Metadata directory not provided
I20260620 09:28:44.231784 31046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:28:44.233151 31046 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:28:44.233794 31065 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:28:44.233951 31046 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 09:28:44.234051 31046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-1/wal
uuid: "52358e0cc7da4808a747d83dab337afd"
format_stamp: "Formatted at 2026-06-20 09:28:43 on dist-test-slave-706q"
server_key: "837d3a2056f4b79c44161b683e4c2532"
server_key_iv: "aceb559cad56c1fd490b5a56da04005b"
server_key_version: "encryptionkey@0"
I20260620 09:28:44.234400 31046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:28:44.246673 31046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:28:44.248780 31046 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:28:44.248926 31046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:28:44.249156 31046 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:28:44.249598 31072 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 09:28:44.251030 31046 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 09:28:44.251085 31046 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260620 09:28:44.251114 31046 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 09:28:44.251896 31046 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 09:28:44.251930 31046 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.001s sys 0.000s
I20260620 09:28:44.252022 31072 tablet_bootstrap.cc:492] T d8d96d70e12445f1aa93fdb71d97100b P 52358e0cc7da4808a747d83dab337afd: Bootstrap starting.
I20260620 09:28:44.257799 31046 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.130:38497
I20260620 09:28:44.257915 31179 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.130:38497 every 8 connection(s)
I20260620 09:28:44.258215 31046 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-1/data/info.pb
I20260620 09:28:44.262478 31072 log.cc:824] T d8d96d70e12445f1aa93fdb71d97100b P 52358e0cc7da4808a747d83dab337afd: Log is configured to *not* fsync() on all Append() calls
Jun 20 09:28:44 dist-test-slave-706q krb5kdc[30272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947724, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.130@KRBTEST.COM for kudu/127.16.43.190@KRBTEST.COM
I20260620 09:28:44.265455 31072 tablet_bootstrap.cc:492] T d8d96d70e12445f1aa93fdb71d97100b P 52358e0cc7da4808a747d83dab337afd: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 09:28:44.265770 31072 tablet_bootstrap.cc:492] T d8d96d70e12445f1aa93fdb71d97100b P 52358e0cc7da4808a747d83dab337afd: Bootstrap complete.
I20260620 09:28:44.266270 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 31046
I20260620 09:28:44.266526 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.43.131:33529
--local_ip_for_outbound_sockets=127.16.43.131
--tserver_master_addrs=127.16.43.190:38073
--webserver_port=44601
--webserver_interface=127.16.43.131
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.131.keytab
--principal=kudu/127.16.43.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.43.148:42633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 09:28:44.266944 31072 ts_tablet_manager.cc:1397] T d8d96d70e12445f1aa93fdb71d97100b P 52358e0cc7da4808a747d83dab337afd: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.011s
I20260620 09:28:44.268563 31072 raft_consensus.cc:348] T d8d96d70e12445f1aa93fdb71d97100b P 52358e0cc7da4808a747d83dab337afd [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55de9465487443d0bcd713075a4d8438" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 43333 } } peers { permanent_uuid: "52358e0cc7da4808a747d83dab337afd" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 38497 } } peers { permanent_uuid: "ca9f7e88ebe741259e36b11e1ddedca5" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 33529 } }
I20260620 09:28:44.268880 31072 raft_consensus.cc:729] T d8d96d70e12445f1aa93fdb71d97100b P 52358e0cc7da4808a747d83dab337afd [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52358e0cc7da4808a747d83dab337afd, State: Initialized, Role: FOLLOWER
I20260620 09:28:44.269063 31044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:44.259006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.130:34665 (local address 127.16.43.190:38073)
0620 09:28:44.259079 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0620 09:28:44.259082 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:28:44.259771 (+ 689us) server_negotiation.cc:407] Connection header received
0620 09:28:44.260244 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:28:44.260245 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:28:44.260262 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:28:44.260356 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:28:44.261031 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:28:44.261854 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:28:44.262999 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:28:44.263235 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:28:44.263297 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:28:44.265889 (+ 2592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:28:44.265891 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:28:44.265891 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:28:44.265899 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:28:44.267424 (+ 1525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:28:44.267869 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:28:44.267870 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:28:44.267871 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:28:44.267935 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:28:44.268267 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:28:44.268268 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:28:44.268268 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:28:44.268420 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:28:44.268498 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 09:28:44.268863 (+ 365us) server_negotiation.cc:299] Negotiation successful
0620 09:28:44.268964 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 09:28:44.269021 31072 consensus_queue.cc:260] T d8d96d70e12445f1aa93fdb71d97100b P 52358e0cc7da4808a747d83dab337afd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55de9465487443d0bcd713075a4d8438" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 43333 } } peers { permanent_uuid: "52358e0cc7da4808a747d83dab337afd" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 38497 } } peers { permanent_uuid: "ca9f7e88ebe741259e36b11e1ddedca5" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 33529 } }
I20260620 09:28:44.269300 31182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:44.258921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:38073 (local address 127.16.43.130:34665)
0620 09:28:44.259639 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0620 09:28:44.259676 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 09:28:44.260134 (+ 458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:28:44.260410 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:28:44.260414 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:28:44.260693 (+ 279us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:28:44.260888 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:28:44.260893 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:28:44.261973 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:28:44.261974 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:28:44.262901 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:28:44.262905 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:28:44.263345 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:28:44.263346 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:28:44.263490 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:28:44.264054 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:28:44.264064 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:28:44.265760 (+ 1696us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:28:44.267511 (+ 1751us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:28:44.267519 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:28:44.267521 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:28:44.267778 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:28:44.268015 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:28:44.268016 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:28:44.268017 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:28:44.268171 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:28:44.268495 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:28:44.268501 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:28:44.268689 (+ 188us) client_negotiation.cc:769] Sending connection context
0620 09:28:44.268800 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 09:28:44.269001 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":676,"thread_start_us":110,"threads_started":1}
I20260620 09:28:44.269337 31072 ts_tablet_manager.cc:1428] T d8d96d70e12445f1aa93fdb71d97100b P 52358e0cc7da4808a747d83dab337afd: Time spent starting tablet: real 0.002s user 0.000s sys 0.006s
I20260620 09:28:44.269865 31180 heartbeater.cc:347] Connected to a master server at 127.16.43.190:38073
I20260620 09:28:44.269968 31180 heartbeater.cc:464] Registering TS with master...
I20260620 09:28:44.270157 31180 heartbeater.cc:511] Master 127.16.43.190:38073 requested a full tablet report, sending...
I20260620 09:28:44.270727 30853 ts_manager.cc:194] Registered new tserver with Master: 52358e0cc7da4808a747d83dab337afd (127.16.43.130:38497)
I20260620 09:28:44.271382 30853 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.43.130@KRBTEST.COM'} at 127.16.43.130:34665
I20260620 09:28:44.272351 31180 heartbeater.cc:503] Master 127.16.43.190:38073 was elected leader, sending a full tablet report...
Jun 20 09:28:44 dist-test-slave-706q krb5kdc[30272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947723, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.190@KRBTEST.COM for kudu/127.16.43.130@KRBTEST.COM
I20260620 09:28:44.279141 31048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:44.271027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.130:38497 (local address 127.0.0.1:49102)
0620 09:28:44.271111 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0620 09:28:44.271124 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 09:28:44.271165 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:28:44.271628 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:28:44.271630 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:28:44.271992 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:28:44.272038 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:28:44.272041 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:28:44.273169 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:28:44.273170 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:28:44.273744 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:28:44.273749 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:28:44.274241 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:28:44.274242 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:28:44.274302 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:28:44.274883 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:28:44.274887 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:28:44.276271 (+ 1384us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:28:44.278000 (+ 1729us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:28:44.278008 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:28:44.278010 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:28:44.278216 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:28:44.278430 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:28:44.278431 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:28:44.278431 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:28:44.278491 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:28:44.278773 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:28:44.278781 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:28:44.278832 (+ 51us) client_negotiation.cc:769] Sending connection context
0620 09:28:44.278916 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 09:28:44.279035 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260620 09:28:44.279139 31186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:44.271234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49102 (local address 127.16.43.130:38497)
0620 09:28:44.271376 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 09:28:44.271381 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:28:44.271390 (+ 9us) server_negotiation.cc:407] Connection header received
0620 09:28:44.271427 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:28:44.271429 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:28:44.271460 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:28:44.272119 (+ 659us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:28:44.272212 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:28:44.273095 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:28:44.273836 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:28:44.274152 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:28:44.274240 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:28:44.276360 (+ 2120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:28:44.276368 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:28:44.276369 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:28:44.276376 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:28:44.277909 (+ 1533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:28:44.278299 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:28:44.278302 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:28:44.278302 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:28:44.278363 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:28:44.278557 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:28:44.278558 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:28:44.278559 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:28:44.278704 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:28:44.278797 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 09:28:44.278951 (+ 154us) server_negotiation.cc:299] Negotiation successful
0620 09:28:44.279035 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":114,"thread_start_us":87,"threads_started":1}
I20260620 09:28:44.279618 31114 tablet_service.cc:1514] Processing DeleteTablet for tablet d8d96d70e12445f1aa93fdb71d97100b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:28:43 UTC) from {username='kudu', principal='kudu/127.16.43.190@KRBTEST.COM'} at 127.0.0.1:49102
I20260620 09:28:44.280218 31187 tablet_replica.cc:330] stopping tablet replica
I20260620 09:28:44.280344 31187 raft_consensus.cc:2229] T d8d96d70e12445f1aa93fdb71d97100b P 52358e0cc7da4808a747d83dab337afd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:28:44.280433 31187 raft_consensus.cc:2258] T d8d96d70e12445f1aa93fdb71d97100b P 52358e0cc7da4808a747d83dab337afd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:28:44.280722 31187 ts_tablet_manager.cc:1905] T d8d96d70e12445f1aa93fdb71d97100b P 52358e0cc7da4808a747d83dab337afd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:28:44.282823 31187 ts_tablet_manager.cc:1918] T d8d96d70e12445f1aa93fdb71d97100b P 52358e0cc7da4808a747d83dab337afd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:28:44.282917 31187 log.cc:1196] T d8d96d70e12445f1aa93fdb71d97100b P 52358e0cc7da4808a747d83dab337afd: Deleting WAL directory at /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-1/wal/wals/d8d96d70e12445f1aa93fdb71d97100b
I20260620 09:28:44.283146 31187 ts_tablet_manager.cc:1939] T d8d96d70e12445f1aa93fdb71d97100b P 52358e0cc7da4808a747d83dab337afd: Deleting consensus metadata
I20260620 09:28:44.283747 30840 catalog_manager.cc:4841] TS 52358e0cc7da4808a747d83dab337afd (127.16.43.130:38497): tablet d8d96d70e12445f1aa93fdb71d97100b (table test-table [id=add96c37041c41d8a37bf6f9ca1cb60b]) successfully deleted
W20260620 09:28:44.357192 31184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:28:44.357422 31184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:28:44.357448 31184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:28:44.358652 31184 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:28:44.358688 31184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:28:44.358714 31184 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.43.131
I20260620 09:28:44.360153 31184 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:42633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.131.keytab
--principal=kudu/127.16.43.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.43.131:33529
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.43.131
--webserver_port=44601
--tserver_master_addrs=127.16.43.190:38073
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31184
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.43.131
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:28:44.360344 31184 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:28:44.360538 31184 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 09:28:44.365720 31184 server_base.cc:1034] running on GCE node
W20260620 09:28:44.365651 31192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:28:44.365653 31193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:28:44.365651 31195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:28:44.366050 31184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:28:44.366345 31184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:28:44.367503 31184 hybrid_clock.cc:648] HybridClock initialized: now 1781947724367493 us; error 36 us; skew 500 ppm
Jun 20 09:28:44 dist-test-slave-706q krb5kdc[30272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947724, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:28:44.370021 31184 init.cc:377] Logged in from keytab as kudu/127.16.43.131@KRBTEST.COM (short username kudu)
I20260620 09:28:44.370417 31184 webserver.cc:460] Webserver started at http://127.16.43.131:44601/ using document root <none> and password file <none>
I20260620 09:28:44.370604 31184 fs_manager.cc:359] Metadata directory not provided
I20260620 09:28:44.370656 31184 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:28:44.372068 31184 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 09:28:44.372650 31202 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:28:44.372850 31184 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 09:28:44.372939 31184 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-2/wal
uuid: "ca9f7e88ebe741259e36b11e1ddedca5"
format_stamp: "Formatted at 2026-06-20 09:28:43 on dist-test-slave-706q"
server_key: "e2bf67ce12b0713de4b59bdf0f2819d3"
server_key_iv: "afc58e6e5dd41fb8d0402815203423de"
server_key_version: "encryptionkey@0"
I20260620 09:28:44.373301 31184 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:28:44.407967 31184 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:28:44.410193 31184 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:28:44.410336 31184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:28:44.410588 31184 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:28:44.410905 31184 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:28:44.410938 31184 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:28:44.410964 31184 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:28:44.411006 31184 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:28:44.416967 31184 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.131:33529
I20260620 09:28:44.417011 31315 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.131:33529 every 8 connection(s)
I20260620 09:28:44.417352 31184 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-2/data/info.pb
I20260620 09:28:44.423272 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 31184
Jun 20 09:28:44 dist-test-slave-706q krb5kdc[30272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947724, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.131@KRBTEST.COM for kudu/127.16.43.190@KRBTEST.COM
I20260620 09:28:44.427465 31044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:44.418072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.131:49763 (local address 127.16.43.190:38073)
0620 09:28:44.418144 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0620 09:28:44.418146 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:28:44.418356 (+ 210us) server_negotiation.cc:407] Connection header received
0620 09:28:44.418902 (+ 546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:28:44.418903 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:28:44.418945 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:28:44.419017 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:28:44.420067 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:28:44.420569 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:28:44.421537 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:28:44.421727 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:28:44.421802 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:28:44.424358 (+ 2556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:28:44.424360 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:28:44.424360 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:28:44.424366 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:28:44.425835 (+ 1469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:28:44.426375 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:28:44.426376 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:28:44.426377 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:28:44.426454 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:28:44.426741 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:28:44.426742 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:28:44.426742 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:28:44.426876 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:28:44.426959 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 09:28:44.427258 (+ 299us) server_negotiation.cc:299] Negotiation successful
0620 09:28:44.427347 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 09:28:44.427726 31318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:44.418071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:38073 (local address 127.16.43.131:49763)
0620 09:28:44.418231 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 09:28:44.418269 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 09:28:44.418753 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:28:44.419096 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:28:44.419099 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:28:44.419784 (+ 685us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:28:44.419980 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:28:44.419985 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:28:44.420651 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:28:44.420652 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:28:44.421445 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:28:44.421447 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:28:44.421814 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:28:44.421815 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:28:44.421913 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:28:44.422452 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:28:44.422460 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:28:44.424266 (+ 1806us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:28:44.425909 (+ 1643us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:28:44.425924 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:28:44.425927 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:28:44.426288 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:28:44.426528 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:28:44.426530 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:28:44.426530 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:28:44.426666 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:28:44.426950 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:28:44.426957 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:28:44.427109 (+ 152us) client_negotiation.cc:769] Sending connection context
0620 09:28:44.427239 (+ 130us) client_negotiation.cc:240] Negotiation successful
0620 09:28:44.427416 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":94,"threads_started":1}
I20260620 09:28:44.428277 31316 heartbeater.cc:347] Connected to a master server at 127.16.43.190:38073
I20260620 09:28:44.428390 31316 heartbeater.cc:464] Registering TS with master...
I20260620 09:28:44.428581 31316 heartbeater.cc:511] Master 127.16.43.190:38073 requested a full tablet report, sending...
I20260620 09:28:44.428982 30853 ts_manager.cc:194] Registered new tserver with Master: ca9f7e88ebe741259e36b11e1ddedca5 (127.16.43.131:33529)
I20260620 09:28:44.429517 30853 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.43.131@KRBTEST.COM'} at 127.16.43.131:49763
I20260620 09:28:44.430681 31320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:44.423763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34478 (local address 127.16.43.190:38073)
0620 09:28:44.423870 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0620 09:28:44.423872 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:28:44.423954 (+ 82us) server_negotiation.cc:407] Connection header received
0620 09:28:44.424057 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:28:44.424057 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:28:44.424067 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:28:44.424124 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:28:44.424780 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:28:44.425289 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:28:44.426185 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:28:44.426381 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:28:44.426492 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:28:44.427891 (+ 1399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:28:44.427897 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:28:44.427898 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:28:44.427904 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:28:44.429653 (+ 1749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:28:44.429977 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:28:44.429978 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:28:44.429978 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:28:44.430034 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:28:44.430240 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:28:44.430241 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:28:44.430241 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:28:44.430343 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:28:44.430499 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 09:28:44.430552 (+ 53us) server_negotiation.cc:299] Negotiation successful
0620 09:28:44.430620 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88,"thread_start_us":53,"threads_started":1}
I20260620 09:28:44.431391 16558 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 09:28:44.434536 31320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:44.431747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34486 (local address 127.16.43.190:38073)
0620 09:28:44.431807 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 09:28:44.431809 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:28:44.431898 (+ 89us) server_negotiation.cc:407] Connection header received
0620 09:28:44.431982 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:28:44.431983 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:28:44.432022 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:28:44.432117 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:28:44.432212 (+ 95us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:28:44.432914 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:28:44.433737 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:28:44.433949 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:28:44.434022 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:28:44.434204 (+ 182us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:28:44.434307 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:28:44.434393 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 09:28:44.434432 (+ 39us) server_negotiation.cc:299] Negotiation successful
0620 09:28:44.434456 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 09:28:44.434804 30853 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:34486:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 09:28:44.436480 30853 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 09:28:44.438597 30975 tablet_service.cc:1467] Processing CreateTablet for tablet d2edfa3efac946e1ab50d6dd89f67083 (DEFAULT_TABLE table=test-table [id=74a888321bc741c3b4841b0eae8c021a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:28:44.438627 31114 tablet_service.cc:1467] Processing CreateTablet for tablet d2edfa3efac946e1ab50d6dd89f67083 (DEFAULT_TABLE table=test-table [id=74a888321bc741c3b4841b0eae8c021a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:28:44.438807 31114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2edfa3efac946e1ab50d6dd89f67083. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:28:44.438807 30975 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2edfa3efac946e1ab50d6dd89f67083. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:28:44.440387 30933 tablet_bootstrap.cc:492] T d2edfa3efac946e1ab50d6dd89f67083 P 55de9465487443d0bcd713075a4d8438: Bootstrap starting.
I20260620 09:28:44.440716 30933 tablet_bootstrap.cc:654] T d2edfa3efac946e1ab50d6dd89f67083 P 55de9465487443d0bcd713075a4d8438: Neither blocks nor log segments found. Creating new log.
I20260620 09:28:44.440950 31072 tablet_bootstrap.cc:492] T d2edfa3efac946e1ab50d6dd89f67083 P 52358e0cc7da4808a747d83dab337afd: Bootstrap starting.
I20260620 09:28:44.441255 31072 tablet_bootstrap.cc:654] T d2edfa3efac946e1ab50d6dd89f67083 P 52358e0cc7da4808a747d83dab337afd: Neither blocks nor log segments found. Creating new log.
I20260620 09:28:44.441282 30933 tablet_bootstrap.cc:492] T d2edfa3efac946e1ab50d6dd89f67083 P 55de9465487443d0bcd713075a4d8438: No bootstrap required, opened a new log
I20260620 09:28:44.441322 30933 ts_tablet_manager.cc:1397] T d2edfa3efac946e1ab50d6dd89f67083 P 55de9465487443d0bcd713075a4d8438: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260620 09:28:44.441447 30933 raft_consensus.cc:348] T d2edfa3efac946e1ab50d6dd89f67083 P 55de9465487443d0bcd713075a4d8438 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca9f7e88ebe741259e36b11e1ddedca5" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 33529 } } peers { permanent_uuid: "52358e0cc7da4808a747d83dab337afd" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 38497 } } peers { permanent_uuid: "55de9465487443d0bcd713075a4d8438" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 43333 } }
I20260620 09:28:44.441550 30933 raft_consensus.cc:374] T d2edfa3efac946e1ab50d6dd89f67083 P 55de9465487443d0bcd713075a4d8438 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:28:44.441614 30933 raft_consensus.cc:729] T d2edfa3efac946e1ab50d6dd89f67083 P 55de9465487443d0bcd713075a4d8438 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55de9465487443d0bcd713075a4d8438, State: Initialized, Role: FOLLOWER
I20260620 09:28:44.441675 30933 consensus_queue.cc:260] T d2edfa3efac946e1ab50d6dd89f67083 P 55de9465487443d0bcd713075a4d8438 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca9f7e88ebe741259e36b11e1ddedca5" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 33529 } } peers { permanent_uuid: "52358e0cc7da4808a747d83dab337afd" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 38497 } } peers { permanent_uuid: "55de9465487443d0bcd713075a4d8438" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 43333 } }
I20260620 09:28:44.441781 30933 ts_tablet_manager.cc:1428] T d2edfa3efac946e1ab50d6dd89f67083 P 55de9465487443d0bcd713075a4d8438: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260620 09:28:44.442099 31072 tablet_bootstrap.cc:492] T d2edfa3efac946e1ab50d6dd89f67083 P 52358e0cc7da4808a747d83dab337afd: No bootstrap required, opened a new log
I20260620 09:28:44.442167 31072 ts_tablet_manager.cc:1397] T d2edfa3efac946e1ab50d6dd89f67083 P 52358e0cc7da4808a747d83dab337afd: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260620 09:28:44.442314 31072 raft_consensus.cc:348] T d2edfa3efac946e1ab50d6dd89f67083 P 52358e0cc7da4808a747d83dab337afd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca9f7e88ebe741259e36b11e1ddedca5" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 33529 } } peers { permanent_uuid: "52358e0cc7da4808a747d83dab337afd" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 38497 } } peers { permanent_uuid: "55de9465487443d0bcd713075a4d8438" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 43333 } }
I20260620 09:28:44.442392 31072 raft_consensus.cc:374] T d2edfa3efac946e1ab50d6dd89f67083 P 52358e0cc7da4808a747d83dab337afd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:28:44.442440 31072 raft_consensus.cc:729] T d2edfa3efac946e1ab50d6dd89f67083 P 52358e0cc7da4808a747d83dab337afd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52358e0cc7da4808a747d83dab337afd, State: Initialized, Role: FOLLOWER
I20260620 09:28:44.442492 31072 consensus_queue.cc:260] T d2edfa3efac946e1ab50d6dd89f67083 P 52358e0cc7da4808a747d83dab337afd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca9f7e88ebe741259e36b11e1ddedca5" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 33529 } } peers { permanent_uuid: "52358e0cc7da4808a747d83dab337afd" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 38497 } } peers { permanent_uuid: "55de9465487443d0bcd713075a4d8438" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 43333 } }
I20260620 09:28:44.442577 31072 ts_tablet_manager.cc:1428] T d2edfa3efac946e1ab50d6dd89f67083 P 52358e0cc7da4808a747d83dab337afd: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260620 09:28:44.442989 31048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:44.438383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.131:33529 (local address 127.0.0.1:57340)
0620 09:28:44.438420 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0620 09:28:44.438432 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 09:28:44.438481 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:28:44.438870 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:28:44.438871 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:28:44.438873 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:28:44.438920 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:28:44.438922 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:28:44.439961 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:28:44.439962 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:28:44.440874 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:28:44.440877 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:28:44.442753 (+ 1876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:28:44.442754 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:28:44.442821 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:28:44.442825 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 09:28:44.442884 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 09:28:44.442924 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19}
I20260620 09:28:44.443027 31322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:44.438477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57340 (local address 127.16.43.131:33529)
0620 09:28:44.438695 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0620 09:28:44.438701 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 09:28:44.438711 (+ 10us) server_negotiation.cc:407] Connection header received
0620 09:28:44.438751 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:28:44.438753 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:28:44.438768 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:28:44.438994 (+ 226us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:28:44.439055 (+ 61us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:28:44.439867 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:28:44.441817 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:28:44.442568 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:28:44.442755 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:28:44.442828 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 09:28:44.442898 (+ 70us) server_negotiation.cc:299] Negotiation successful
0620 09:28:44.442938 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":143,"threads_started":1}
I20260620 09:28:44.443778 31250 tablet_service.cc:1467] Processing CreateTablet for tablet d2edfa3efac946e1ab50d6dd89f67083 (DEFAULT_TABLE table=test-table [id=74a888321bc741c3b4841b0eae8c021a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:28:44.444095 31250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2edfa3efac946e1ab50d6dd89f67083. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:28:44.446843 31325 tablet_bootstrap.cc:492] T d2edfa3efac946e1ab50d6dd89f67083 P ca9f7e88ebe741259e36b11e1ddedca5: Bootstrap starting.
I20260620 09:28:44.447440 31325 tablet_bootstrap.cc:654] T d2edfa3efac946e1ab50d6dd89f67083 P ca9f7e88ebe741259e36b11e1ddedca5: Neither blocks nor log segments found. Creating new log.
I20260620 09:28:44.447942 31325 log.cc:824] T d2edfa3efac946e1ab50d6dd89f67083 P ca9f7e88ebe741259e36b11e1ddedca5: Log is configured to *not* fsync() on all Append() calls
I20260620 09:28:44.448745 31325 tablet_bootstrap.cc:492] T d2edfa3efac946e1ab50d6dd89f67083 P ca9f7e88ebe741259e36b11e1ddedca5: No bootstrap required, opened a new log
I20260620 09:28:44.448818 31325 ts_tablet_manager.cc:1397] T d2edfa3efac946e1ab50d6dd89f67083 P ca9f7e88ebe741259e36b11e1ddedca5: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 09:28:44.450117 31325 raft_consensus.cc:348] T d2edfa3efac946e1ab50d6dd89f67083 P ca9f7e88ebe741259e36b11e1ddedca5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca9f7e88ebe741259e36b11e1ddedca5" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 33529 } } peers { permanent_uuid: "52358e0cc7da4808a747d83dab337afd" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 38497 } } peers { permanent_uuid: "55de9465487443d0bcd713075a4d8438" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 43333 } }
I20260620 09:28:44.450251 31325 raft_consensus.cc:374] T d2edfa3efac946e1ab50d6dd89f67083 P ca9f7e88ebe741259e36b11e1ddedca5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:28:44.450273 31325 raft_consensus.cc:729] T d2edfa3efac946e1ab50d6dd89f67083 P ca9f7e88ebe741259e36b11e1ddedca5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca9f7e88ebe741259e36b11e1ddedca5, State: Initialized, Role: FOLLOWER
I20260620 09:28:44.450390 31325 consensus_queue.cc:260] T d2edfa3efac946e1ab50d6dd89f67083 P ca9f7e88ebe741259e36b11e1ddedca5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca9f7e88ebe741259e36b11e1ddedca5" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 33529 } } peers { permanent_uuid: "52358e0cc7da4808a747d83dab337afd" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 38497 } } peers { permanent_uuid: "55de9465487443d0bcd713075a4d8438" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 43333 } }
I20260620 09:28:44.450691 31316 heartbeater.cc:503] Master 127.16.43.190:38073 was elected leader, sending a full tablet report...
I20260620 09:28:44.450696 31325 ts_tablet_manager.cc:1428] T d2edfa3efac946e1ab50d6dd89f67083 P ca9f7e88ebe741259e36b11e1ddedca5: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
W20260620 09:28:44.508822 31181 tablet.cc:2369] T d2edfa3efac946e1ab50d6dd89f67083 P 52358e0cc7da4808a747d83dab337afd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 09:28:44.545383 31047 raft_consensus.cc:482] T d2edfa3efac946e1ab50d6dd89f67083 P 55de9465487443d0bcd713075a4d8438 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:28:44.545526 31047 raft_consensus.cc:504] T d2edfa3efac946e1ab50d6dd89f67083 P 55de9465487443d0bcd713075a4d8438 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca9f7e88ebe741259e36b11e1ddedca5" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 33529 } } peers { permanent_uuid: "52358e0cc7da4808a747d83dab337afd" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 38497 } } peers { permanent_uuid: "55de9465487443d0bcd713075a4d8438" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 43333 } }
I20260620 09:28:44.545925 31047 leader_election.cc:288] T d2edfa3efac946e1ab50d6dd89f67083 P 55de9465487443d0bcd713075a4d8438 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ca9f7e88ebe741259e36b11e1ddedca5 (127.16.43.131:33529), 52358e0cc7da4808a747d83dab337afd (127.16.43.130:38497)
I20260620 09:28:44.549325 31322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:44.546239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.129:60493 (local address 127.16.43.131:33529)
0620 09:28:44.546285 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 09:28:44.546287 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:28:44.546299 (+ 12us) server_negotiation.cc:407] Connection header received
0620 09:28:44.546370 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:28:44.546371 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:28:44.546380 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:28:44.546433 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:28:44.546599 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:28:44.547329 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:28:44.548464 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:28:44.548989 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:28:44.549045 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:28:44.549052 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0620 09:28:44.549243 (+ 191us) server_negotiation.cc:299] Negotiation successful
0620 09:28:44.549271 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 09:28:44.549374 31043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:44.546152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.131:33529 (local address 127.16.43.129:60493)
0620 09:28:44.546221 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0620 09:28:44.546236 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 09:28:44.546306 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:28:44.546449 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:28:44.546450 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:28:44.546452 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:28:44.546524 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:28:44.546526 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:28:44.547409 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:28:44.547410 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:28:44.548371 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:28:44.548373 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:28:44.549079 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:28:44.549080 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:28:44.549157 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:28:44.549161 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 09:28:44.549238 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 09:28:44.549280 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260620 09:28:44.549806 31328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:44.546271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.130:38497 (local address 127.16.43.129:38775)
0620 09:28:44.546430 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 09:28:44.546439 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 09:28:44.546488 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:28:44.546675 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:28:44.546676 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:28:44.546678 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:28:44.547056 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:28:44.547058 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:28:44.547885 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:28:44.547885 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:28:44.548768 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:28:44.548770 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:28:44.549561 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:28:44.549562 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:28:44.549635 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:28:44.549643 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 09:28:44.549705 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 09:28:44.549743 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"mutex_wait_us":1,"thread_start_us":101,"threads_started":1}
I20260620 09:28:44.549826 31186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:44.546345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.129:38775 (local address 127.16.43.130:38497)
0620 09:28:44.546393 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 09:28:44.546395 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:28:44.546502 (+ 107us) server_negotiation.cc:407] Connection header received
0620 09:28:44.546552 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:28:44.546553 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:28:44.546585 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:28:44.546651 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:28:44.547141 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:28:44.547806 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:28:44.548876 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:28:44.549480 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:28:44.549554 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:28:44.549654 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 09:28:44.549710 (+ 56us) server_negotiation.cc:299] Negotiation successful
0620 09:28:44.549763 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 09:28:44.549815 31270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2edfa3efac946e1ab50d6dd89f67083" candidate_uuid: "55de9465487443d0bcd713075a4d8438" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca9f7e88ebe741259e36b11e1ddedca5" is_pre_election: true
I20260620 09:28:44.549993 31270 raft_consensus.cc:2454] T d2edfa3efac946e1ab50d6dd89f67083 P ca9f7e88ebe741259e36b11e1ddedca5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 55de9465487443d0bcd713075a4d8438 in term 0.
I20260620 09:28:44.550248 30930 leader_election.cc:302] T d2edfa3efac946e1ab50d6dd89f67083 P 55de9465487443d0bcd713075a4d8438 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55de9465487443d0bcd713075a4d8438, ca9f7e88ebe741259e36b11e1ddedca5; no voters:
I20260620 09:28:44.550323 31134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2edfa3efac946e1ab50d6dd89f67083" candidate_uuid: "55de9465487443d0bcd713075a4d8438" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52358e0cc7da4808a747d83dab337afd" is_pre_election: true
I20260620 09:28:44.550398 31047 raft_consensus.cc:2781] T d2edfa3efac946e1ab50d6dd89f67083 P 55de9465487443d0bcd713075a4d8438 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 09:28:44.550472 31047 raft_consensus.cc:482] T d2edfa3efac946e1ab50d6dd89f67083 P 55de9465487443d0bcd713075a4d8438 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 09:28:44.550482 31134 raft_consensus.cc:2454] T d2edfa3efac946e1ab50d6dd89f67083 P 52358e0cc7da4808a747d83dab337afd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 55de9465487443d0bcd713075a4d8438 in term 0.
I20260620 09:28:44.550506 31047 raft_consensus.cc:3037] T d2edfa3efac946e1ab50d6dd89f67083 P 55de9465487443d0bcd713075a4d8438 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:28:44.551226 31047 raft_consensus.cc:504] T d2edfa3efac946e1ab50d6dd89f67083 P 55de9465487443d0bcd713075a4d8438 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca9f7e88ebe741259e36b11e1ddedca5" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 33529 } } peers { permanent_uuid: "52358e0cc7da4808a747d83dab337afd" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 38497 } } peers { permanent_uuid: "55de9465487443d0bcd713075a4d8438" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 43333 } }
I20260620 09:28:44.551369 31047 leader_election.cc:288] T d2edfa3efac946e1ab50d6dd89f67083 P 55de9465487443d0bcd713075a4d8438 [CANDIDATE]: Term 1 election: Requested vote from peers ca9f7e88ebe741259e36b11e1ddedca5 (127.16.43.131:33529), 52358e0cc7da4808a747d83dab337afd (127.16.43.130:38497)
I20260620 09:28:44.551515 31270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2edfa3efac946e1ab50d6dd89f67083" candidate_uuid: "55de9465487443d0bcd713075a4d8438" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca9f7e88ebe741259e36b11e1ddedca5"
I20260620 09:28:44.551594 31134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2edfa3efac946e1ab50d6dd89f67083" candidate_uuid: "55de9465487443d0bcd713075a4d8438" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52358e0cc7da4808a747d83dab337afd"
I20260620 09:28:44.551651 31270 raft_consensus.cc:3037] T d2edfa3efac946e1ab50d6dd89f67083 P ca9f7e88ebe741259e36b11e1ddedca5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:28:44.551678 31134 raft_consensus.cc:3037] T d2edfa3efac946e1ab50d6dd89f67083 P 52358e0cc7da4808a747d83dab337afd [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:28:44.552358 31134 raft_consensus.cc:2454] T d2edfa3efac946e1ab50d6dd89f67083 P 52358e0cc7da4808a747d83dab337afd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55de9465487443d0bcd713075a4d8438 in term 1.
I20260620 09:28:44.552380 31270 raft_consensus.cc:2454] T d2edfa3efac946e1ab50d6dd89f67083 P ca9f7e88ebe741259e36b11e1ddedca5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55de9465487443d0bcd713075a4d8438 in term 1.
I20260620 09:28:44.552539 30930 leader_election.cc:302] T d2edfa3efac946e1ab50d6dd89f67083 P 55de9465487443d0bcd713075a4d8438 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55de9465487443d0bcd713075a4d8438, ca9f7e88ebe741259e36b11e1ddedca5; no voters:
I20260620 09:28:44.552664 31047 raft_consensus.cc:2781] T d2edfa3efac946e1ab50d6dd89f67083 P 55de9465487443d0bcd713075a4d8438 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:28:44.552852 31047 raft_consensus.cc:686] T d2edfa3efac946e1ab50d6dd89f67083 P 55de9465487443d0bcd713075a4d8438 [term 1 LEADER]: Becoming Leader. State: Replica: 55de9465487443d0bcd713075a4d8438, State: Running, Role: LEADER
I20260620 09:28:44.552915 31047 consensus_queue.cc:237] T d2edfa3efac946e1ab50d6dd89f67083 P 55de9465487443d0bcd713075a4d8438 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca9f7e88ebe741259e36b11e1ddedca5" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 33529 } } peers { permanent_uuid: "52358e0cc7da4808a747d83dab337afd" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 38497 } } peers { permanent_uuid: "55de9465487443d0bcd713075a4d8438" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 43333 } }
I20260620 09:28:44.553784 30852 catalog_manager.cc:5495] T d2edfa3efac946e1ab50d6dd89f67083 P 55de9465487443d0bcd713075a4d8438 reported cstate change: term changed from 0 to 1, leader changed from <none> to 55de9465487443d0bcd713075a4d8438 (127.16.43.129). New cstate: current_term: 1 leader_uuid: "55de9465487443d0bcd713075a4d8438" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca9f7e88ebe741259e36b11e1ddedca5" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 33529 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52358e0cc7da4808a747d83dab337afd" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 38497 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "55de9465487443d0bcd713075a4d8438" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 43333 } health_report { overall_health: HEALTHY } } }
I20260620 09:28:44.560619 31049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:44.558138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44008 (local address 127.16.43.129:43333)
0620 09:28:44.558183 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 09:28:44.558185 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:28:44.558195 (+ 10us) server_negotiation.cc:407] Connection header received
0620 09:28:44.558265 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:28:44.558266 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:28:44.558285 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:28:44.558375 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:28:44.558484 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:28:44.558998 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:28:44.559837 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:28:44.560043 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:28:44.560109 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:28:44.560262 (+ 153us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:28:44.560378 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:28:44.560470 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 09:28:44.560511 (+ 41us) server_negotiation.cc:299] Negotiation successful
0620 09:28:44.560533 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 09:28:44.562211 31270 raft_consensus.cc:1261] T d2edfa3efac946e1ab50d6dd89f67083 P ca9f7e88ebe741259e36b11e1ddedca5 [term 1 FOLLOWER]: Refusing update from remote peer 55de9465487443d0bcd713075a4d8438: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:28:44.562229 31134 raft_consensus.cc:1261] T d2edfa3efac946e1ab50d6dd89f67083 P 52358e0cc7da4808a747d83dab337afd [term 1 FOLLOWER]: Refusing update from remote peer 55de9465487443d0bcd713075a4d8438: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:28:44.562690 31329 consensus_queue.cc:1035] T d2edfa3efac946e1ab50d6dd89f67083 P 55de9465487443d0bcd713075a4d8438 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca9f7e88ebe741259e36b11e1ddedca5" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 33529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:28:44.562829 31047 consensus_queue.cc:1035] T d2edfa3efac946e1ab50d6dd89f67083 P 55de9465487443d0bcd713075a4d8438 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52358e0cc7da4808a747d83dab337afd" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 38497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:28:44.565836 31333 mvcc.cc:205] Tried to move back new op lower bound from 7298857879803977728 to 7298857879769276416. Current Snapshot: MvccSnapshot[applied={T|T < 7298857879803977728}]
I20260620 09:28:44.565613 31334 mvcc.cc:205] Tried to move back new op lower bound from 7298857879803977728 to 7298857879769276416. Current Snapshot: MvccSnapshot[applied={T|T < 7298857879803977728}]
I20260620 09:28:44.567302 30853 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:34486:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:28:44.567405 30853 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:34486:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:28:44.568179 30853 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 516727c9072f4e0f8034238b71a751d3: Sending DeleteTablet for 3 replicas of tablet d2edfa3efac946e1ab50d6dd89f67083
I20260620 09:28:44.568461 30975 tablet_service.cc:1514] Processing DeleteTablet for tablet d2edfa3efac946e1ab50d6dd89f67083 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:28:44 UTC) from {username='kudu', principal='kudu/127.16.43.190@KRBTEST.COM'} at 127.0.0.1:43996
I20260620 09:28:44.568483 31250 tablet_service.cc:1514] Processing DeleteTablet for tablet d2edfa3efac946e1ab50d6dd89f67083 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:28:44 UTC) from {username='kudu', principal='kudu/127.16.43.190@KRBTEST.COM'} at 127.0.0.1:57340
I20260620 09:28:44.568526 31114 tablet_service.cc:1514] Processing DeleteTablet for tablet d2edfa3efac946e1ab50d6dd89f67083 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:28:44 UTC) from {username='kudu', principal='kudu/127.16.43.190@KRBTEST.COM'} at 127.0.0.1:49102
I20260620 09:28:44.568554 31050 tablet_replica.cc:330] T d2edfa3efac946e1ab50d6dd89f67083 P 55de9465487443d0bcd713075a4d8438: stopping tablet replica
I20260620 09:28:44.568553 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 30891
I20260620 09:28:44.568601 31050 raft_consensus.cc:2229] T d2edfa3efac946e1ab50d6dd89f67083 P 55de9465487443d0bcd713075a4d8438 [term 1 LEADER]: Raft consensus shutting down.
I20260620 09:28:44.568671 31187 tablet_replica.cc:330] T d2edfa3efac946e1ab50d6dd89f67083 P 52358e0cc7da4808a747d83dab337afd: stopping tablet replica
I20260620 09:28:44.568684 31050 raft_consensus.cc:2258] T d2edfa3efac946e1ab50d6dd89f67083 P 55de9465487443d0bcd713075a4d8438 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:28:44.568729 31187 raft_consensus.cc:2229] T d2edfa3efac946e1ab50d6dd89f67083 P 52358e0cc7da4808a747d83dab337afd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:28:44.568787 31187 raft_consensus.cc:2258] T d2edfa3efac946e1ab50d6dd89f67083 P 52358e0cc7da4808a747d83dab337afd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:28:44.569388 31187 ts_tablet_manager.cc:1905] T d2edfa3efac946e1ab50d6dd89f67083 P 52358e0cc7da4808a747d83dab337afd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:28:44.569487 31342 tablet_replica.cc:330] T d2edfa3efac946e1ab50d6dd89f67083 P ca9f7e88ebe741259e36b11e1ddedca5: stopping tablet replica
I20260620 09:28:44.569589 31342 raft_consensus.cc:2229] T d2edfa3efac946e1ab50d6dd89f67083 P ca9f7e88ebe741259e36b11e1ddedca5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:28:44.569705 31342 raft_consensus.cc:2258] T d2edfa3efac946e1ab50d6dd89f67083 P ca9f7e88ebe741259e36b11e1ddedca5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:28:44.570156 31342 ts_tablet_manager.cc:1905] T d2edfa3efac946e1ab50d6dd89f67083 P ca9f7e88ebe741259e36b11e1ddedca5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:28:44.571139 31187 ts_tablet_manager.cc:1918] T d2edfa3efac946e1ab50d6dd89f67083 P 52358e0cc7da4808a747d83dab337afd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:28:44.571192 31187 log.cc:1196] T d2edfa3efac946e1ab50d6dd89f67083 P 52358e0cc7da4808a747d83dab337afd: Deleting WAL directory at /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-1/wal/wals/d2edfa3efac946e1ab50d6dd89f67083
I20260620 09:28:44.571414 31187 ts_tablet_manager.cc:1939] T d2edfa3efac946e1ab50d6dd89f67083 P 52358e0cc7da4808a747d83dab337afd: Deleting consensus metadata
I20260620 09:28:44.571928 30840 catalog_manager.cc:4841] TS 52358e0cc7da4808a747d83dab337afd (127.16.43.130:38497): tablet d2edfa3efac946e1ab50d6dd89f67083 (table test-table [id=74a888321bc741c3b4841b0eae8c021a]) successfully deleted
I20260620 09:28:44.572167 31342 ts_tablet_manager.cc:1918] T d2edfa3efac946e1ab50d6dd89f67083 P ca9f7e88ebe741259e36b11e1ddedca5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:28:44.572224 31342 log.cc:1196] T d2edfa3efac946e1ab50d6dd89f67083 P ca9f7e88ebe741259e36b11e1ddedca5: Deleting WAL directory at /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-2/wal/wals/d2edfa3efac946e1ab50d6dd89f67083
I20260620 09:28:44.572503 31342 ts_tablet_manager.cc:1939] T d2edfa3efac946e1ab50d6dd89f67083 P ca9f7e88ebe741259e36b11e1ddedca5: Deleting consensus metadata
I20260620 09:28:44.572994 30840 catalog_manager.cc:4841] TS ca9f7e88ebe741259e36b11e1ddedca5 (127.16.43.131:33529): tablet d2edfa3efac946e1ab50d6dd89f67083 (table test-table [id=74a888321bc741c3b4841b0eae8c021a]) successfully deleted
W20260620 09:28:44.575356 30837 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.43.129:43333 (error 108)
I20260620 09:28:44.575943 31048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:44.575804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.129:43333 (local address 127.0.0.1:44018)
0620 09:28:44.575848 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0620 09:28:44.575888 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.129:43333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260620 09:28:44.576124 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 31046
W20260620 09:28:44.576089 30837 catalog_manager.cc:4584] TS 55de9465487443d0bcd713075a4d8438 (127.16.43.129:43333): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d2edfa3efac946e1ab50d6dd89f67083: Network error: Client connection negotiation failed: client connection to 127.16.43.129:43333: connect: Connection refused (error 111)
I20260620 09:28:44.581777 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 31184
I20260620 09:28:44.587499 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 30820
I20260620 09:28:44.596896 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.43.190:38073
--webserver_interface=127.16.43.190
--webserver_port=43503
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.190.keytab
--principal=kudu/127.16.43.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.43.148:42633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.43.190:38073
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:28:44.678597 31344 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:28:44.678828 31344 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:28:44.678853 31344 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:28:44.680018 31344 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:28:44.680056 31344 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:28:44.680070 31344 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:28:44.680083 31344 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:28:44.680094 31344 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:28:44.681546 31344 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:42633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.43.190:38073
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.190.keytab
--principal=kudu/127.16.43.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.43.190:38073
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.43.190
--webserver_port=43503
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31344
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:28:44.681749 31344 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:28:44.681937 31344 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:28:44.686933 31349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:28:44.686926 31352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:28:44.686926 31350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:28:44.687248 31344 server_base.cc:1034] running on GCE node
I20260620 09:28:44.687520 31344 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:28:44.687920 31344 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:28:44.689090 31344 hybrid_clock.cc:648] HybridClock initialized: now 1781947724689082 us; error 37 us; skew 500 ppm
Jun 20 09:28:44 dist-test-slave-706q krb5kdc[30272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947724, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:28:44.691784 31344 init.cc:377] Logged in from keytab as kudu/127.16.43.190@KRBTEST.COM (short username kudu)
I20260620 09:28:44.692349 31344 webserver.cc:460] Webserver started at http://127.16.43.190:43503/ using document root <none> and password file <none>
I20260620 09:28:44.692601 31344 fs_manager.cc:359] Metadata directory not provided
I20260620 09:28:44.692672 31344 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:28:44.694073 31344 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:28:44.694880 31359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:28:44.695101 31344 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:28:44.695206 31344 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/master-0/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/master-0/wal
uuid: "516727c9072f4e0f8034238b71a751d3"
format_stamp: "Formatted at 2026-06-20 09:28:43 on dist-test-slave-706q"
server_key: "1bf6a9adabdb7e214fbfe067da2bb3f6"
server_key_iv: "bc521b0d457322537acc73bfbfeec5ae"
server_key_version: "encryptionkey@0"
I20260620 09:28:44.695609 31344 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:28:44.709874 31344 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:28:44.712050 31344 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:28:44.712170 31344 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:28:44.716377 31344 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.190:38073
I20260620 09:28:44.716408 31411 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.190:38073 every 8 connection(s)
I20260620 09:28:44.716800 31344 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
I20260620 09:28:44.718430 31412 sys_catalog.cc:263] Verifying existing consensus state
I20260620 09:28:44.719132 31412 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 516727c9072f4e0f8034238b71a751d3: Bootstrap starting.
I20260620 09:28:44.721695 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 31344
I20260620 09:28:44.721994 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.43.129:43333
--local_ip_for_outbound_sockets=127.16.43.129
--tserver_master_addrs=127.16.43.190:38073
--webserver_port=42289
--webserver_interface=127.16.43.129
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.129.keytab
--principal=kudu/127.16.43.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.43.148:42633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 09:28:44.728014 31412 log.cc:824] T 00000000000000000000000000000000 P 516727c9072f4e0f8034238b71a751d3: Log is configured to *not* fsync() on all Append() calls
I20260620 09:28:44.730316 31412 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 516727c9072f4e0f8034238b71a751d3: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 09:28:44.730638 31412 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 516727c9072f4e0f8034238b71a751d3: Bootstrap complete.
I20260620 09:28:44.732957 31412 raft_consensus.cc:348] T 00000000000000000000000000000000 P 516727c9072f4e0f8034238b71a751d3 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "516727c9072f4e0f8034238b71a751d3" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 38073 } }
I20260620 09:28:44.733209 31412 raft_consensus.cc:729] T 00000000000000000000000000000000 P 516727c9072f4e0f8034238b71a751d3 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 516727c9072f4e0f8034238b71a751d3, State: Initialized, Role: FOLLOWER
I20260620 09:28:44.733402 31412 consensus_queue.cc:260] T 00000000000000000000000000000000 P 516727c9072f4e0f8034238b71a751d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "516727c9072f4e0f8034238b71a751d3" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 38073 } }
I20260620 09:28:44.733496 31412 raft_consensus.cc:388] T 00000000000000000000000000000000 P 516727c9072f4e0f8034238b71a751d3 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:28:44.733549 31412 raft_consensus.cc:482] T 00000000000000000000000000000000 P 516727c9072f4e0f8034238b71a751d3 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:28:44.733605 31412 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 516727c9072f4e0f8034238b71a751d3 [term 2 FOLLOWER]: Advancing to term 3
I20260620 09:28:44.734592 31412 raft_consensus.cc:504] T 00000000000000000000000000000000 P 516727c9072f4e0f8034238b71a751d3 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "516727c9072f4e0f8034238b71a751d3" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 38073 } }
I20260620 09:28:44.734720 31412 leader_election.cc:302] T 00000000000000000000000000000000 P 516727c9072f4e0f8034238b71a751d3 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 516727c9072f4e0f8034238b71a751d3; no voters:
I20260620 09:28:44.734910 31412 leader_election.cc:288] T 00000000000000000000000000000000 P 516727c9072f4e0f8034238b71a751d3 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 09:28:44.734985 31416 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 516727c9072f4e0f8034238b71a751d3 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 09:28:44.735183 31412 sys_catalog.cc:564] T 00000000000000000000000000000000 P 516727c9072f4e0f8034238b71a751d3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:28:44.735194 31416 raft_consensus.cc:686] T 00000000000000000000000000000000 P 516727c9072f4e0f8034238b71a751d3 [term 3 LEADER]: Becoming Leader. State: Replica: 516727c9072f4e0f8034238b71a751d3, State: Running, Role: LEADER
I20260620 09:28:44.735426 31416 consensus_queue.cc:237] T 00000000000000000000000000000000 P 516727c9072f4e0f8034238b71a751d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "516727c9072f4e0f8034238b71a751d3" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 38073 } }
I20260620 09:28:44.735728 31417 sys_catalog.cc:455] T 00000000000000000000000000000000 P 516727c9072f4e0f8034238b71a751d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "516727c9072f4e0f8034238b71a751d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "516727c9072f4e0f8034238b71a751d3" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 38073 } } }
I20260620 09:28:44.735798 31417 sys_catalog.cc:458] T 00000000000000000000000000000000 P 516727c9072f4e0f8034238b71a751d3 [sys.catalog]: This master's current role is: LEADER
I20260620 09:28:44.735916 31418 sys_catalog.cc:455] T 00000000000000000000000000000000 P 516727c9072f4e0f8034238b71a751d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 516727c9072f4e0f8034238b71a751d3. Latest consensus state: current_term: 3 leader_uuid: "516727c9072f4e0f8034238b71a751d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "516727c9072f4e0f8034238b71a751d3" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 38073 } } }
I20260620 09:28:44.735965 31418 sys_catalog.cc:458] T 00000000000000000000000000000000 P 516727c9072f4e0f8034238b71a751d3 [sys.catalog]: This master's current role is: LEADER
I20260620 09:28:44.736060 31424 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:28:44.737042 31424 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:28:44.737175 31424 catalog_manager.cc:1259] Loaded cluster ID: 8b0481e1fb5546209dc536267159980d
I20260620 09:28:44.737213 31424 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260620 09:28:44.737601 31424 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f9d552c69ad google::LogMessage::Fail() at ??:0
@ 0x7f9d552c988a google::LogMessage::SendToLog() at ??:0
@ 0x7f9d552c64de google::LogMessage::Flush() at ??:0
@ 0x7f9d552c7a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f9d5d1e70f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f9d5d1f08b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f9d571940d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f9d57189954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f9d57fed6db start_thread at ??:0
@ 0x7f9d53dc461f clone at ??:0
W20260620 09:28:44.846617 31414 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:28:44.846957 31414 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:28:44.847004 31414 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:28:44.848889 31414 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:28:44.848956 31414 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:28:44.849000 31414 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.43.129
I20260620 09:28:44.851194 31414 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:42633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.129.keytab
--principal=kudu/127.16.43.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.43.129:43333
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.43.129
--webserver_port=42289
--tserver_master_addrs=127.16.43.190:38073
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31414
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.43.129
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:28:44.851509 31414 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:28:44.851859 31414 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:28:44.857573 31451 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:28:44.857592 31448 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:28:44.857584 31449 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:28:44.857908 31414 server_base.cc:1034] running on GCE node
I20260620 09:28:44.858170 31414 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:28:44.858457 31414 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:28:44.859594 31414 hybrid_clock.cc:648] HybridClock initialized: now 1781947724859577 us; error 33 us; skew 500 ppm
Jun 20 09:28:44 dist-test-slave-706q krb5kdc[30272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947724, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:28:44.862176 31414 init.cc:377] Logged in from keytab as kudu/127.16.43.129@KRBTEST.COM (short username kudu)
I20260620 09:28:44.862672 31414 webserver.cc:460] Webserver started at http://127.16.43.129:42289/ using document root <none> and password file <none>
I20260620 09:28:44.862862 31414 fs_manager.cc:359] Metadata directory not provided
I20260620 09:28:44.862942 31414 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:28:44.864401 31414 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 09:28:44.865159 31458 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:28:44.865350 31414 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 09:28:44.865444 31414 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-0/wal
uuid: "55de9465487443d0bcd713075a4d8438"
format_stamp: "Formatted at 2026-06-20 09:28:43 on dist-test-slave-706q"
server_key: "b8cf4182d8252833daa52bc821848630"
server_key_iv: "1d181cee9ea91d529ae9d23154d68401"
server_key_version: "encryptionkey@0"
I20260620 09:28:44.865816 31414 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:28:44.873384 31414 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:28:44.875792 31414 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:28:44.875943 31414 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:28:44.876186 31414 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:28:44.876636 31465 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 09:28:44.878611 31414 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 09:28:44.878693 31414 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260620 09:28:44.878743 31414 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 09:28:44.879418 31414 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 09:28:44.879479 31414 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260620 09:28:44.879587 31465 tablet_bootstrap.cc:492] T d2edfa3efac946e1ab50d6dd89f67083 P 55de9465487443d0bcd713075a4d8438: Bootstrap starting.
I20260620 09:28:44.886386 31414 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.129:43333
I20260620 09:28:44.886520 31572 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.129:43333 every 8 connection(s)
I20260620 09:28:44.886878 31414 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-0/data/info.pb
I20260620 09:28:44.887705 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 31414
I20260620 09:28:44.888006 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.43.130:38497
--local_ip_for_outbound_sockets=127.16.43.130
--tserver_master_addrs=127.16.43.190:38073
--webserver_port=36893
--webserver_interface=127.16.43.130
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.130.keytab
--principal=kudu/127.16.43.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.43.148:42633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 09:28:44.890676 31465 log.cc:824] T d2edfa3efac946e1ab50d6dd89f67083 P 55de9465487443d0bcd713075a4d8438: Log is configured to *not* fsync() on all Append() calls
I20260620 09:28:44.893141 31465 tablet_bootstrap.cc:492] T d2edfa3efac946e1ab50d6dd89f67083 P 55de9465487443d0bcd713075a4d8438: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 09:28:44.893432 31465 tablet_bootstrap.cc:492] T d2edfa3efac946e1ab50d6dd89f67083 P 55de9465487443d0bcd713075a4d8438: Bootstrap complete.
I20260620 09:28:44.894376 31465 ts_tablet_manager.cc:1397] T d2edfa3efac946e1ab50d6dd89f67083 P 55de9465487443d0bcd713075a4d8438: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.009s
I20260620 09:28:44.895499 31465 raft_consensus.cc:348] T d2edfa3efac946e1ab50d6dd89f67083 P 55de9465487443d0bcd713075a4d8438 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca9f7e88ebe741259e36b11e1ddedca5" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 33529 } } peers { permanent_uuid: "52358e0cc7da4808a747d83dab337afd" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 38497 } } peers { permanent_uuid: "55de9465487443d0bcd713075a4d8438" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 43333 } }
I20260620 09:28:44.895833 31465 raft_consensus.cc:729] T d2edfa3efac946e1ab50d6dd89f67083 P 55de9465487443d0bcd713075a4d8438 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55de9465487443d0bcd713075a4d8438, State: Initialized, Role: FOLLOWER
I20260620 09:28:44.896030 31465 consensus_queue.cc:260] T d2edfa3efac946e1ab50d6dd89f67083 P 55de9465487443d0bcd713075a4d8438 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca9f7e88ebe741259e36b11e1ddedca5" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 33529 } } peers { permanent_uuid: "52358e0cc7da4808a747d83dab337afd" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 38497 } } peers { permanent_uuid: "55de9465487443d0bcd713075a4d8438" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 43333 } }
I20260620 09:28:44.896325 31465 ts_tablet_manager.cc:1428] T d2edfa3efac946e1ab50d6dd89f67083 P 55de9465487443d0bcd713075a4d8438: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
W20260620 09:28:44.991801 31576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:28:44.992026 31576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:28:44.992053 31576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:28:44.993386 31576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:28:44.993444 31576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:28:44.993474 31576 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.43.130
I20260620 09:28:44.994990 31576 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:42633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.130.keytab
--principal=kudu/127.16.43.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.43.130:38497
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.43.130
--webserver_port=36893
--tserver_master_addrs=127.16.43.190:38073
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31576
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.43.130
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:28:44.995251 31576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:28:44.995512 31576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:28:45.000757 31583 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:28:45.000907 31584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:28:45.000757 31586 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:28:45.000968 31576 server_base.cc:1034] running on GCE node
I20260620 09:28:45.001351 31576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:28:45.001647 31576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:28:45.002820 31576 hybrid_clock.cc:648] HybridClock initialized: now 1781947725002806 us; error 29 us; skew 500 ppm
Jun 20 09:28:45 dist-test-slave-706q krb5kdc[30272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947725, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:28:45.005715 31576 init.cc:377] Logged in from keytab as kudu/127.16.43.130@KRBTEST.COM (short username kudu)
I20260620 09:28:45.006283 31576 webserver.cc:460] Webserver started at http://127.16.43.130:36893/ using document root <none> and password file <none>
I20260620 09:28:45.006532 31576 fs_manager.cc:359] Metadata directory not provided
I20260620 09:28:45.006580 31576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:28:45.008468 31576 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:28:45.009227 31593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:28:45.009407 31576 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:28:45.009493 31576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-1/wal
uuid: "52358e0cc7da4808a747d83dab337afd"
format_stamp: "Formatted at 2026-06-20 09:28:43 on dist-test-slave-706q"
server_key: "837d3a2056f4b79c44161b683e4c2532"
server_key_iv: "aceb559cad56c1fd490b5a56da04005b"
server_key_version: "encryptionkey@0"
I20260620 09:28:45.009989 31576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20260620 09:28:45.012439 31575 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:28:44.887718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:38073 (local address 127.16.43.129:38355)
0620 09:28:44.887888 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 09:28:44.887931 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 09:28:44.888609 (+ 678us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:28:45.012245 (+123636us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:38073: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":98,"threads_started":1}
W20260620 09:28:45.012730 31573 heartbeater.cc:644] Failed to heartbeat to 127.16.43.190:38073 (0 consecutive failures): Network error: Failed to ping master at 127.16.43.190:38073: Client connection negotiation failed: client connection to 127.16.43.190:38073: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 09:28:45.013070 31575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:45.012954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:38073 (local address 127.16.43.129:45427)
0620 09:28:45.013006 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0620 09:28:45.013050 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:38073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260620 09:28:45.013415 31575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:45.013336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:38073 (local address 127.16.43.129:46997)
0620 09:28:45.013378 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0620 09:28:45.013401 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:38073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260620 09:28:45.013537 31573 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 09:28:45.035807 31576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:28:45.037750 31576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:28:45.037904 31576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:28:45.038159 31576 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:28:45.038477 31576 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:28:45.038529 31576 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:28:45.038571 31576 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:28:45.038600 31576 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:28:45.044569 31576 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.130:38497
I20260620 09:28:45.044611 31706 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.130:38497 every 8 connection(s)
I20260620 09:28:45.044958 31576 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-1/data/info.pb
I20260620 09:28:45.046008 31709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:45.045588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:38073 (local address 127.16.43.130:39729)
0620 09:28:45.045738 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 09:28:45.045854 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:38073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":70,"threads_started":1}
W20260620 09:28:45.046411 31707 heartbeater.cc:644] Failed to heartbeat to 127.16.43.190:38073 (0 consecutive failures): Network error: Failed to ping master at 127.16.43.190:38073: Client connection negotiation failed: client connection to 127.16.43.190:38073: connect: Connection refused (error 111)
I20260620 09:28:45.046707 31709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:45.046613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:38073 (local address 127.16.43.130:38481)
0620 09:28:45.046662 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0620 09:28:45.046689 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:38073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260620 09:28:45.046990 31709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:45.046919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:38073 (local address 127.16.43.130:47307)
0620 09:28:45.046956 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0620 09:28:45.046977 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:38073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260620 09:28:45.047127 31707 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 09:28:45.055068 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 31576
I20260620 09:28:45.055269 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.43.131:33529
--local_ip_for_outbound_sockets=127.16.43.131
--tserver_master_addrs=127.16.43.190:38073
--webserver_port=44601
--webserver_interface=127.16.43.131
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.131.keytab
--principal=kudu/127.16.43.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.43.148:42633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:28:45.136245 31710 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:28:45.136507 31710 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:28:45.136547 31710 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:28:45.137785 31710 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:28:45.137866 31710 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:28:45.137910 31710 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.43.131
I20260620 09:28:45.139459 31710 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:42633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.131.keytab
--principal=kudu/127.16.43.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.43.131:33529
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.43.131
--webserver_port=44601
--tserver_master_addrs=127.16.43.190:38073
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31710
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.43.131
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:28:45.139773 31710 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:28:45.140017 31710 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:28:45.145215 31715 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:28:45.145284 31716 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:28:45.145224 31718 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:28:45.145529 31710 server_base.cc:1034] running on GCE node
I20260620 09:28:45.145787 31710 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:28:45.146045 31710 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:28:45.147215 31710 hybrid_clock.cc:648] HybridClock initialized: now 1781947725147205 us; error 30 us; skew 500 ppm
Jun 20 09:28:45 dist-test-slave-706q krb5kdc[30272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947725, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:28:45.149761 31710 init.cc:377] Logged in from keytab as kudu/127.16.43.131@KRBTEST.COM (short username kudu)
I20260620 09:28:45.150235 31710 webserver.cc:460] Webserver started at http://127.16.43.131:44601/ using document root <none> and password file <none>
I20260620 09:28:45.150465 31710 fs_manager.cc:359] Metadata directory not provided
I20260620 09:28:45.150530 31710 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:28:45.151928 31710 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:28:45.152516 31725 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:28:45.152695 31710 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 09:28:45.152796 31710 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-2/wal
uuid: "ca9f7e88ebe741259e36b11e1ddedca5"
format_stamp: "Formatted at 2026-06-20 09:28:43 on dist-test-slave-706q"
server_key: "e2bf67ce12b0713de4b59bdf0f2819d3"
server_key_iv: "afc58e6e5dd41fb8d0402815203423de"
server_key_version: "encryptionkey@0"
I20260620 09:28:45.153133 31710 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:28:45.170599 31710 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:28:45.172645 31710 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:28:45.172809 31710 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:28:45.173048 31710 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:28:45.173362 31710 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:28:45.173414 31710 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:28:45.173455 31710 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:28:45.173483 31710 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:28:45.180193 31710 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.131:33529
I20260620 09:28:45.180233 31838 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.131:33529 every 8 connection(s)
I20260620 09:28:45.180828 31710 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781947562949448-16558-0/minicluster-data/ts-2/data/info.pb
I20260620 09:28:45.181988 31841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:45.181561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:38073 (local address 127.16.43.131:48111)
0620 09:28:45.181714 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 09:28:45.181834 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:38073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":72,"threads_started":1}
W20260620 09:28:45.182432 31839 heartbeater.cc:644] Failed to heartbeat to 127.16.43.190:38073 (0 consecutive failures): Network error: Failed to ping master at 127.16.43.190:38073: Client connection negotiation failed: client connection to 127.16.43.190:38073: connect: Connection refused (error 111)
I20260620 09:28:45.182723 31841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:45.182621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:38073 (local address 127.16.43.131:54847)
0620 09:28:45.182677 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0620 09:28:45.182705 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:38073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260620 09:28:45.183002 31841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:45.182923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:38073 (local address 127.16.43.131:53467)
0620 09:28:45.182967 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0620 09:28:45.182988 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:38073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260620 09:28:45.183122 31839 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 09:28:45.190209 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 31710
I20260620 09:28:46.014230 31843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:46.013946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:38073 (local address 127.16.43.129:33997)
0620 09:28:46.014094 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 09:28:46.014156 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:38073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":65,"threads_started":1}
I20260620 09:28:46.047727 31844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:46.047443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:38073 (local address 127.16.43.130:58037)
0620 09:28:46.047634 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0620 09:28:46.047676 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:38073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":57,"threads_started":1}
I20260620 09:28:46.183857 31845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:46.183614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:38073 (local address 127.16.43.131:39963)
0620 09:28:46.183760 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 09:28:46.183801 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:38073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":60,"threads_started":1}
I20260620 09:28:46.534686 31846 raft_consensus.cc:482] T d2edfa3efac946e1ab50d6dd89f67083 P 55de9465487443d0bcd713075a4d8438 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:28:46.534812 31846 raft_consensus.cc:504] T d2edfa3efac946e1ab50d6dd89f67083 P 55de9465487443d0bcd713075a4d8438 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca9f7e88ebe741259e36b11e1ddedca5" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 33529 } } peers { permanent_uuid: "52358e0cc7da4808a747d83dab337afd" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 38497 } } peers { permanent_uuid: "55de9465487443d0bcd713075a4d8438" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 43333 } }
I20260620 09:28:46.535208 31846 leader_election.cc:288] T d2edfa3efac946e1ab50d6dd89f67083 P 55de9465487443d0bcd713075a4d8438 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ca9f7e88ebe741259e36b11e1ddedca5 (127.16.43.131:33529), 52358e0cc7da4808a747d83dab337afd (127.16.43.130:38497)
Jun 20 09:28:46 dist-test-slave-706q krb5kdc[30272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947724, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.129@KRBTEST.COM for kudu/127.16.43.131@KRBTEST.COM
Jun 20 09:28:46 dist-test-slave-706q krb5kdc[30272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947724, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.129@KRBTEST.COM for kudu/127.16.43.130@KRBTEST.COM
I20260620 09:28:46.547073 31849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:46.535699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.130:38497 (local address 127.16.43.129:39525)
0620 09:28:46.535795 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0620 09:28:46.535804 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 09:28:46.535855 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:28:46.536904 (+ 1049us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:28:46.536905 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:28:46.537177 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:28:46.537374 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:28:46.537380 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:28:46.538468 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:28:46.538469 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:28:46.539277 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:28:46.539279 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:28:46.539956 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:28:46.539957 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:28:46.540071 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:28:46.540707 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:28:46.540712 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:28:46.542547 (+ 1835us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:28:46.544866 (+ 2319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:28:46.544876 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:28:46.544878 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:28:46.545330 (+ 452us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:28:46.545653 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:28:46.545654 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:28:46.545655 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:28:46.545835 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:28:46.546448 (+ 613us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:28:46.546449 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:28:46.546599 (+ 150us) client_negotiation.cc:769] Sending connection context
0620 09:28:46.546730 (+ 131us) client_negotiation.cc:240] Negotiation successful
0620 09:28:46.546947 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85,"mutex_wait_us":70,"thread_start_us":50,"threads_started":1}
I20260620 09:28:46.547168 31847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:46.535449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.131:33529 (local address 127.16.43.129:47045)
0620 09:28:46.535571 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0620 09:28:46.535583 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 09:28:46.535647 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:28:46.536758 (+ 1111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:28:46.536761 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:28:46.537179 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:28:46.537374 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:28:46.537381 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:28:46.538459 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:28:46.538460 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:28:46.539277 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:28:46.539280 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:28:46.539959 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:28:46.539959 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:28:46.540081 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:28:46.540672 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:28:46.540690 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:28:46.542303 (+ 1613us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:28:46.544866 (+ 2563us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:28:46.544876 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:28:46.544878 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:28:46.545330 (+ 452us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:28:46.545674 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:28:46.545675 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:28:46.545676 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:28:46.545835 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:28:46.546375 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:28:46.546387 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:28:46.546599 (+ 212us) client_negotiation.cc:769] Sending connection context
0620 09:28:46.546729 (+ 130us) client_negotiation.cc:240] Negotiation successful
0620 09:28:46.547022 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"mutex_wait_us":188,"thread_start_us":50,"threads_started":1}
I20260620 09:28:46.547283 31848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:46.535571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.129:47045 (local address 127.16.43.131:33529)
0620 09:28:46.535723 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 09:28:46.535735 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 09:28:46.535774 (+ 39us) server_negotiation.cc:407] Connection header received
0620 09:28:46.536443 (+ 669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:28:46.536447 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:28:46.536508 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:28:46.536690 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:28:46.537498 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:28:46.538350 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:28:46.539369 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:28:46.539857 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:28:46.539993 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:28:46.542472 (+ 2479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:28:46.542478 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:28:46.542478 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:28:46.542497 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:28:46.544740 (+ 2243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:28:46.545424 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:28:46.545427 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:28:46.545427 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:28:46.545541 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:28:46.545917 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:28:46.545918 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:28:46.545918 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:28:46.546258 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:28:46.546348 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 09:28:46.546990 (+ 642us) server_negotiation.cc:299] Negotiation successful
0620 09:28:46.547151 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":88,"threads_started":1}
I20260620 09:28:46.547662 31850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:46.535847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.129:39525 (local address 127.16.43.130:38497)
0620 09:28:46.535983 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 09:28:46.535992 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 09:28:46.536061 (+ 69us) server_negotiation.cc:407] Connection header received
0620 09:28:46.536722 (+ 661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:28:46.536725 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:28:46.536774 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:28:46.536913 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:28:46.537495 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:28:46.538351 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:28:46.539369 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:28:46.539858 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:28:46.539980 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:28:46.542656 (+ 2676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:28:46.542660 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:28:46.542660 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:28:46.542672 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:28:46.544745 (+ 2073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:28:46.545423 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:28:46.545427 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:28:46.545427 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:28:46.545542 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:28:46.545917 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:28:46.545918 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:28:46.545918 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:28:46.546253 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:28:46.546443 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0620 09:28:46.546800 (+ 357us) server_negotiation.cc:299] Negotiation successful
0620 09:28:46.547521 (+ 721us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":65,"threads_started":1}
W20260620 09:28:46.548112 31462 leader_election.cc:341] T d2edfa3efac946e1ab50d6dd89f67083 P 55de9465487443d0bcd713075a4d8438 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ca9f7e88ebe741259e36b11e1ddedca5 (127.16.43.131:33529): Not found: Tablet not found: d2edfa3efac946e1ab50d6dd89f67083
W20260620 09:28:46.548625 31462 leader_election.cc:341] T d2edfa3efac946e1ab50d6dd89f67083 P 55de9465487443d0bcd713075a4d8438 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 52358e0cc7da4808a747d83dab337afd (127.16.43.130:38497): Not found: Tablet not found: d2edfa3efac946e1ab50d6dd89f67083
I20260620 09:28:46.548683 31462 leader_election.cc:302] T d2edfa3efac946e1ab50d6dd89f67083 P 55de9465487443d0bcd713075a4d8438 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 55de9465487443d0bcd713075a4d8438; no voters: 52358e0cc7da4808a747d83dab337afd, ca9f7e88ebe741259e36b11e1ddedca5
I20260620 09:28:46.548805 31846 raft_consensus.cc:2726] T d2edfa3efac946e1ab50d6dd89f67083 P 55de9465487443d0bcd713075a4d8438 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 09:28:47.014873 31849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:47.014740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:38073 (local address 127.16.43.129:60051)
0620 09:28:47.014798 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0620 09:28:47.014832 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:38073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260620 09:28:47.048360 31851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:47.048171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:38073 (local address 127.16.43.130:36355)
0620 09:28:47.048282 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0620 09:28:47.048318 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:38073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":52,"threads_started":1}
I20260620 09:28:47.184525 31852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:47.184326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:38073 (local address 127.16.43.131:58075)
0620 09:28:47.184449 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0620 09:28:47.184479 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:38073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":59,"threads_started":1}
I20260620 09:28:48.015622 31853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:48.015393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:38073 (local address 127.16.43.129:35649)
0620 09:28:48.015541 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 09:28:48.015576 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:38073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":62,"threads_started":1}
I20260620 09:28:48.049146 31854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:48.048927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:38073 (local address 127.16.43.130:43347)
0620 09:28:48.049055 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0620 09:28:48.049103 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:38073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":61,"threads_started":1}
I20260620 09:28:48.185274 31855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:48.185027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:38073 (local address 127.16.43.131:50829)
0620 09:28:48.185192 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 09:28:48.185228 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:38073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":78,"threads_started":1}
I20260620 09:28:48.673923 31856 raft_consensus.cc:482] T d2edfa3efac946e1ab50d6dd89f67083 P 55de9465487443d0bcd713075a4d8438 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:28:48.674034 31856 raft_consensus.cc:504] T d2edfa3efac946e1ab50d6dd89f67083 P 55de9465487443d0bcd713075a4d8438 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca9f7e88ebe741259e36b11e1ddedca5" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 33529 } } peers { permanent_uuid: "52358e0cc7da4808a747d83dab337afd" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 38497 } } peers { permanent_uuid: "55de9465487443d0bcd713075a4d8438" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 43333 } }
I20260620 09:28:48.674225 31856 leader_election.cc:288] T d2edfa3efac946e1ab50d6dd89f67083 P 55de9465487443d0bcd713075a4d8438 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ca9f7e88ebe741259e36b11e1ddedca5 (127.16.43.131:33529), 52358e0cc7da4808a747d83dab337afd (127.16.43.130:38497)
W20260620 09:28:48.674647 31462 leader_election.cc:341] T d2edfa3efac946e1ab50d6dd89f67083 P 55de9465487443d0bcd713075a4d8438 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 52358e0cc7da4808a747d83dab337afd (127.16.43.130:38497): Not found: Tablet not found: d2edfa3efac946e1ab50d6dd89f67083
W20260620 09:28:48.674741 31462 leader_election.cc:341] T d2edfa3efac946e1ab50d6dd89f67083 P 55de9465487443d0bcd713075a4d8438 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ca9f7e88ebe741259e36b11e1ddedca5 (127.16.43.131:33529): Not found: Tablet not found: d2edfa3efac946e1ab50d6dd89f67083
I20260620 09:28:48.674777 31462 leader_election.cc:302] T d2edfa3efac946e1ab50d6dd89f67083 P 55de9465487443d0bcd713075a4d8438 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 55de9465487443d0bcd713075a4d8438; no voters: 52358e0cc7da4808a747d83dab337afd, ca9f7e88ebe741259e36b11e1ddedca5
I20260620 09:28:48.674854 31856 raft_consensus.cc:2726] T d2edfa3efac946e1ab50d6dd89f67083 P 55de9465487443d0bcd713075a4d8438 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 09:28:49.016464 31857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:49.016186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:38073 (local address 127.16.43.129:56563)
0620 09:28:49.016337 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 09:28:49.016398 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:38073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":71,"threads_started":1}
I20260620 09:28:49.049913 31858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:49.049671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:38073 (local address 127.16.43.130:60605)
0620 09:28:49.049821 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 09:28:49.049867 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:38073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":64,"threads_started":1}
I20260620 09:28:49.186170 31859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:49.185947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:38073 (local address 127.16.43.131:56659)
0620 09:28:49.186085 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 09:28:49.186126 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:38073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":64,"threads_started":1}
I20260620 09:28:50.017376 31860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:50.017103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:38073 (local address 127.16.43.129:53733)
0620 09:28:50.017268 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 09:28:50.017310 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:38073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":70,"threads_started":1}
I20260620 09:28:50.050789 31861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:50.050526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:38073 (local address 127.16.43.130:49789)
0620 09:28:50.050690 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 09:28:50.050734 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:38073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":71,"threads_started":1}
I20260620 09:28:50.187018 31862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:50.186775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:38073 (local address 127.16.43.131:47681)
0620 09:28:50.186937 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 09:28:50.186970 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:38073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":93,"threads_started":1}
I20260620 09:28:50.451822 31863 raft_consensus.cc:482] T d2edfa3efac946e1ab50d6dd89f67083 P 55de9465487443d0bcd713075a4d8438 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:28:50.451937 31863 raft_consensus.cc:504] T d2edfa3efac946e1ab50d6dd89f67083 P 55de9465487443d0bcd713075a4d8438 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca9f7e88ebe741259e36b11e1ddedca5" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 33529 } } peers { permanent_uuid: "52358e0cc7da4808a747d83dab337afd" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 38497 } } peers { permanent_uuid: "55de9465487443d0bcd713075a4d8438" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 43333 } }
I20260620 09:28:50.452126 31863 leader_election.cc:288] T d2edfa3efac946e1ab50d6dd89f67083 P 55de9465487443d0bcd713075a4d8438 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ca9f7e88ebe741259e36b11e1ddedca5 (127.16.43.131:33529), 52358e0cc7da4808a747d83dab337afd (127.16.43.130:38497)
W20260620 09:28:50.452780 31462 leader_election.cc:341] T d2edfa3efac946e1ab50d6dd89f67083 P 55de9465487443d0bcd713075a4d8438 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 52358e0cc7da4808a747d83dab337afd (127.16.43.130:38497): Not found: Tablet not found: d2edfa3efac946e1ab50d6dd89f67083
W20260620 09:28:50.452848 31462 leader_election.cc:341] T d2edfa3efac946e1ab50d6dd89f67083 P 55de9465487443d0bcd713075a4d8438 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ca9f7e88ebe741259e36b11e1ddedca5 (127.16.43.131:33529): Not found: Tablet not found: d2edfa3efac946e1ab50d6dd89f67083
I20260620 09:28:50.452903 31462 leader_election.cc:302] T d2edfa3efac946e1ab50d6dd89f67083 P 55de9465487443d0bcd713075a4d8438 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 55de9465487443d0bcd713075a4d8438; no voters: 52358e0cc7da4808a747d83dab337afd, ca9f7e88ebe741259e36b11e1ddedca5
I20260620 09:28:50.452999 31863 raft_consensus.cc:2726] T d2edfa3efac946e1ab50d6dd89f67083 P 55de9465487443d0bcd713075a4d8438 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 09:28:51.018265 31864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:51.017972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:38073 (local address 127.16.43.129:33171)
0620 09:28:51.018152 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0620 09:28:51.018215 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:38073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":63,"threads_started":1}
I20260620 09:28:51.051687 31865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:51.051385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:38073 (local address 127.16.43.130:59925)
0620 09:28:51.051587 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0620 09:28:51.051637 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:38073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":61,"threads_started":1}
I20260620 09:28:51.187813 31866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:51.187476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:38073 (local address 127.16.43.131:47435)
0620 09:28:51.187729 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0620 09:28:51.187766 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:38073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":187,"threads_started":1}
I20260620 09:28:52.019081 31867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:52.018843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:38073 (local address 127.16.43.129:59135)
0620 09:28:52.018987 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 09:28:52.019027 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:38073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":96,"threads_started":1}
I20260620 09:28:52.052455 31868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:52.052229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:38073 (local address 127.16.43.130:36837)
0620 09:28:52.052364 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0620 09:28:52.052410 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:38073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":62,"threads_started":1}
I20260620 09:28:52.188484 31869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:52.188277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:38073 (local address 127.16.43.131:40051)
0620 09:28:52.188402 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0620 09:28:52.188438 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:38073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":60,"threads_started":1}
I20260620 09:28:53.020028 31870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:53.019771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:38073 (local address 127.16.43.129:44909)
0620 09:28:53.019936 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 09:28:53.019981 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:38073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":66,"threads_started":1}
I20260620 09:28:53.053332 31871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:53.053080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:38073 (local address 127.16.43.130:56783)
0620 09:28:53.053237 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 09:28:53.053284 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:38073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":61,"threads_started":1}
I20260620 09:28:53.189268 31872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:53.189048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:38073 (local address 127.16.43.131:45109)
0620 09:28:53.189179 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0620 09:28:53.189221 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:38073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":65,"threads_started":1}
I20260620 09:28:54.020924 31873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:54.020646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:38073 (local address 127.16.43.129:51359)
0620 09:28:54.020840 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0620 09:28:54.020879 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:38073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":135,"threads_started":1}
I20260620 09:28:54.053987 31874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:54.053794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:38073 (local address 127.16.43.130:48927)
0620 09:28:54.053910 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0620 09:28:54.053942 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:38073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":58,"threads_started":1}
I20260620 09:28:54.190059 31875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:54.189804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:38073 (local address 127.16.43.131:49175)
0620 09:28:54.189963 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 09:28:54.190010 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:38073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":75,"threads_started":1}
I20260620 09:28:54.475510 31876 raft_consensus.cc:482] T d2edfa3efac946e1ab50d6dd89f67083 P 55de9465487443d0bcd713075a4d8438 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:28:54.475692 31876 raft_consensus.cc:504] T d2edfa3efac946e1ab50d6dd89f67083 P 55de9465487443d0bcd713075a4d8438 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca9f7e88ebe741259e36b11e1ddedca5" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 33529 } } peers { permanent_uuid: "52358e0cc7da4808a747d83dab337afd" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 38497 } } peers { permanent_uuid: "55de9465487443d0bcd713075a4d8438" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 43333 } }
I20260620 09:28:54.475875 31876 leader_election.cc:288] T d2edfa3efac946e1ab50d6dd89f67083 P 55de9465487443d0bcd713075a4d8438 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ca9f7e88ebe741259e36b11e1ddedca5 (127.16.43.131:33529), 52358e0cc7da4808a747d83dab337afd (127.16.43.130:38497)
W20260620 09:28:54.476406 31462 leader_election.cc:341] T d2edfa3efac946e1ab50d6dd89f67083 P 55de9465487443d0bcd713075a4d8438 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ca9f7e88ebe741259e36b11e1ddedca5 (127.16.43.131:33529): Not found: Tablet not found: d2edfa3efac946e1ab50d6dd89f67083
W20260620 09:28:54.476500 31462 leader_election.cc:341] T d2edfa3efac946e1ab50d6dd89f67083 P 55de9465487443d0bcd713075a4d8438 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 52358e0cc7da4808a747d83dab337afd (127.16.43.130:38497): Not found: Tablet not found: d2edfa3efac946e1ab50d6dd89f67083
I20260620 09:28:54.476536 31462 leader_election.cc:302] T d2edfa3efac946e1ab50d6dd89f67083 P 55de9465487443d0bcd713075a4d8438 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 55de9465487443d0bcd713075a4d8438; no voters: 52358e0cc7da4808a747d83dab337afd, ca9f7e88ebe741259e36b11e1ddedca5
I20260620 09:28:54.476692 31876 raft_consensus.cc:2726] T d2edfa3efac946e1ab50d6dd89f67083 P 55de9465487443d0bcd713075a4d8438 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 09:28:55.021704 31877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:55.021413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:38073 (local address 127.16.43.129:51801)
0620 09:28:55.021561 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 09:28:55.021600 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:38073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":62,"threads_started":1}
I20260620 09:28:55.054844 31878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:55.054607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:38073 (local address 127.16.43.130:55127)
0620 09:28:55.054764 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 09:28:55.054798 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:38073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":104,"threads_started":1}
I20260620 09:28:55.190788 31879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:55.190587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:38073 (local address 127.16.43.131:40037)
0620 09:28:55.190709 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0620 09:28:55.190741 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:38073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":73,"threads_started":1}
I20260620 09:28:56.022393 31880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:56.022178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:38073 (local address 127.16.43.129:52961)
0620 09:28:56.022315 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 09:28:56.022349 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:38073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":68,"threads_started":1}
I20260620 09:28:56.055521 31881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:56.055303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:38073 (local address 127.16.43.130:49471)
0620 09:28:56.055439 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0620 09:28:56.055476 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:38073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":74,"threads_started":1}
I20260620 09:28:56.191486 31882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:56.191270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:38073 (local address 127.16.43.131:45615)
0620 09:28:56.191406 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0620 09:28:56.191439 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:38073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":75,"threads_started":1}
I20260620 09:28:57.023154 31883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:57.022902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:38073 (local address 127.16.43.129:44087)
0620 09:28:57.023064 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 09:28:57.023107 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:38073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":82,"threads_started":1}
I20260620 09:28:57.056274 31884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:57.056039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:38073 (local address 127.16.43.130:32771)
0620 09:28:57.056172 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 09:28:57.056214 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:38073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":72,"threads_started":1}
I20260620 09:28:57.192370 31885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:57.192115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:38073 (local address 127.16.43.131:45967)
0620 09:28:57.192279 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 09:28:57.192321 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:38073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":79,"threads_started":1}
I20260620 09:28:58.024194 31886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:58.023924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:38073 (local address 127.16.43.129:33379)
0620 09:28:58.024087 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 09:28:58.024132 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:38073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":108,"threads_started":1}
I20260620 09:28:58.056965 31887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:58.056744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:38073 (local address 127.16.43.130:45673)
0620 09:28:58.056881 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 09:28:58.056919 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:38073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":58,"threads_started":1}
I20260620 09:28:58.193181 31888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:58.192930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:38073 (local address 127.16.43.131:45013)
0620 09:28:58.193090 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 09:28:58.193132 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:38073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":93,"threads_started":1}
I20260620 09:28:59.025007 31889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:59.024767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:38073 (local address 127.16.43.129:54145)
0620 09:28:59.024916 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 09:28:59.024958 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:38073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":75,"threads_started":1}
I20260620 09:28:59.057740 31890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:59.057460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:38073 (local address 127.16.43.130:40441)
0620 09:28:59.057601 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 09:28:59.057692 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:38073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":84,"threads_started":1}
I20260620 09:28:59.193930 31891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:28:59.193700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:38073 (local address 127.16.43.131:42521)
0620 09:28:59.193841 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 09:28:59.193881 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:38073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":65,"threads_started":1}
I20260620 09:29:00.026347 31892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:29:00.026030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:38073 (local address 127.16.43.129:41119)
0620 09:29:00.026238 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0620 09:29:00.026289 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:38073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":95,"threads_started":1}
I20260620 09:29:00.058597 31893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:29:00.058311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:38073 (local address 127.16.43.130:35683)
0620 09:29:00.058492 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0620 09:29:00.058546 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:38073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":75,"threads_started":1}
I20260620 09:29:00.194851 31894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:29:00.194551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:38073 (local address 127.16.43.131:38391)
0620 09:29:00.194748 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0620 09:29:00.194799 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:38073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":96,"threads_started":1}
I20260620 09:29:00.198439 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 31414
I20260620 09:29:00.203445 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 31576
I20260620 09:29:00.207751 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 31710
2026-06-20T09:29:00Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18466 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:29:00 dist-test-slave-706q krb5kdc[31897](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:29:00 dist-test-slave-706q krb5kdc[31897](info): set up 2 sockets
Jun 20 09:29:00 dist-test-slave-706q krb5kdc[31897](info): commencing operation
krb5kdc: starting...
W20260620 09:29:01.256636 16558 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/krb5kdc/test-user.keytab.
Jun 20 09:29:01 dist-test-slave-706q krb5kdc[31897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947741, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:29:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:29:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.43.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.43.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.43.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.190.keytab.
Entry for principal kudu/127.16.43.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.190.keytab.
I20260620 09:29:01.387879 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.43.190:44381
--webserver_interface=127.16.43.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.190.keytab
--principal=kudu/127.16.43.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.43.148:41053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.43.190:44381
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:29:01.471015 31910 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:29:01.471246 31910 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:29:01.471272 31910 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:29:01.472572 31910 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 09:29:01.472615 31910 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 09:29:01.472627 31910 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 09:29:01.472638 31910 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:29:01.472651 31910 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:29:01.472664 31910 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:29:01.472677 31910 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:29:01.472726 31910 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:29:01.474331 31910 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:41053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.16.43.190:44381
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.190.keytab
--principal=kudu/127.16.43.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.43.190:44381
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.43.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31910
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:29:01.474586 31910 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:29:01.474824 31910 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:29:01.479925 31916 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:29:01.480019 31915 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:29:01.480057 31910 server_base.cc:1034] running on GCE node
W20260620 09:29:01.479925 31918 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:29:01.480434 31910 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:29:01.480767 31910 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:29:01.481935 31910 hybrid_clock.cc:648] HybridClock initialized: now 1781947741481923 us; error 34 us; skew 500 ppm
Jun 20 09:29:01 dist-test-slave-706q krb5kdc[31897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947741, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:29:01.484750 31910 init.cc:377] Logged in from keytab as kudu/127.16.43.190@KRBTEST.COM (short username kudu)
I20260620 09:29:01.485272 31910 webserver.cc:460] Webserver started at http://127.16.43.190:46375/ using document root <none> and password file <none>
I20260620 09:29:01.485539 31910 fs_manager.cc:359] Metadata directory not provided
I20260620 09:29:01.485606 31910 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:29:01.485761 31910 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:29:01.486919 31910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/master-0/data/instance:
uuid: "c18c7503906044e3b00e5afc76fde6c0"
format_stamp: "Formatted at 2026-06-20 09:29:01 on dist-test-slave-706q"
server_key: "8460fbc73d47b6d32e6e6108117f9d71"
server_key_iv: "bba2020cb7145bbd5a4d8e9b74f35dab"
server_key_version: "encryptionkey@0"
I20260620 09:29:01.487293 31910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/master-0/wal/instance:
uuid: "c18c7503906044e3b00e5afc76fde6c0"
format_stamp: "Formatted at 2026-06-20 09:29:01 on dist-test-slave-706q"
server_key: "8460fbc73d47b6d32e6e6108117f9d71"
server_key_iv: "bba2020cb7145bbd5a4d8e9b74f35dab"
server_key_version: "encryptionkey@0"
I20260620 09:29:01.488782 31910 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.002s
I20260620 09:29:01.489693 31925 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:29:01.489979 31910 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:29:01.490103 31910 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/master-0/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/master-0/wal
uuid: "c18c7503906044e3b00e5afc76fde6c0"
format_stamp: "Formatted at 2026-06-20 09:29:01 on dist-test-slave-706q"
server_key: "8460fbc73d47b6d32e6e6108117f9d71"
server_key_iv: "bba2020cb7145bbd5a4d8e9b74f35dab"
server_key_version: "encryptionkey@0"
I20260620 09:29:01.490199 31910 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:29:01.504910 31910 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:29:01.507381 31910 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:29:01.507659 31910 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:29:01.512480 31910 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.190:44381
I20260620 09:29:01.512558 31977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.190:44381 every 8 connection(s)
I20260620 09:29:01.512969 31910 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
I20260620 09:29:01.513872 31978 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:29:01.516160 31978 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c18c7503906044e3b00e5afc76fde6c0: Bootstrap starting.
I20260620 09:29:01.516726 31978 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c18c7503906044e3b00e5afc76fde6c0: Neither blocks nor log segments found. Creating new log.
I20260620 09:29:01.516971 31978 log.cc:824] T 00000000000000000000000000000000 P c18c7503906044e3b00e5afc76fde6c0: Log is configured to *not* fsync() on all Append() calls
I20260620 09:29:01.517616 31978 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c18c7503906044e3b00e5afc76fde6c0: No bootstrap required, opened a new log
I20260620 09:29:01.519143 31978 raft_consensus.cc:348] T 00000000000000000000000000000000 P c18c7503906044e3b00e5afc76fde6c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c18c7503906044e3b00e5afc76fde6c0" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 44381 } }
I20260620 09:29:01.519305 31978 raft_consensus.cc:374] T 00000000000000000000000000000000 P c18c7503906044e3b00e5afc76fde6c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:29:01.519335 31978 raft_consensus.cc:729] T 00000000000000000000000000000000 P c18c7503906044e3b00e5afc76fde6c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c18c7503906044e3b00e5afc76fde6c0, State: Initialized, Role: FOLLOWER
I20260620 09:29:01.519497 31978 consensus_queue.cc:260] T 00000000000000000000000000000000 P c18c7503906044e3b00e5afc76fde6c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c18c7503906044e3b00e5afc76fde6c0" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 44381 } }
I20260620 09:29:01.519615 31978 raft_consensus.cc:388] T 00000000000000000000000000000000 P c18c7503906044e3b00e5afc76fde6c0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:29:01.519650 31978 raft_consensus.cc:482] T 00000000000000000000000000000000 P c18c7503906044e3b00e5afc76fde6c0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:29:01.519702 31978 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c18c7503906044e3b00e5afc76fde6c0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:29:01.520219 31978 raft_consensus.cc:504] T 00000000000000000000000000000000 P c18c7503906044e3b00e5afc76fde6c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c18c7503906044e3b00e5afc76fde6c0" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 44381 } }
I20260620 09:29:01.520313 31978 leader_election.cc:302] T 00000000000000000000000000000000 P c18c7503906044e3b00e5afc76fde6c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c18c7503906044e3b00e5afc76fde6c0; no voters:
I20260620 09:29:01.520448 31978 leader_election.cc:288] T 00000000000000000000000000000000 P c18c7503906044e3b00e5afc76fde6c0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:29:01.520520 31981 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c18c7503906044e3b00e5afc76fde6c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:29:01.520716 31981 raft_consensus.cc:686] T 00000000000000000000000000000000 P c18c7503906044e3b00e5afc76fde6c0 [term 1 LEADER]: Becoming Leader. State: Replica: c18c7503906044e3b00e5afc76fde6c0, State: Running, Role: LEADER
I20260620 09:29:01.520718 31978 sys_catalog.cc:564] T 00000000000000000000000000000000 P c18c7503906044e3b00e5afc76fde6c0 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:29:01.521116 31981 consensus_queue.cc:237] T 00000000000000000000000000000000 P c18c7503906044e3b00e5afc76fde6c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c18c7503906044e3b00e5afc76fde6c0" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 44381 } }
I20260620 09:29:01.522018 31982 sys_catalog.cc:455] T 00000000000000000000000000000000 P c18c7503906044e3b00e5afc76fde6c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c18c7503906044e3b00e5afc76fde6c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c18c7503906044e3b00e5afc76fde6c0" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 44381 } } }
I20260620 09:29:01.522024 31983 sys_catalog.cc:455] T 00000000000000000000000000000000 P c18c7503906044e3b00e5afc76fde6c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c18c7503906044e3b00e5afc76fde6c0. Latest consensus state: current_term: 1 leader_uuid: "c18c7503906044e3b00e5afc76fde6c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c18c7503906044e3b00e5afc76fde6c0" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 44381 } } }
I20260620 09:29:01.522109 31982 sys_catalog.cc:458] T 00000000000000000000000000000000 P c18c7503906044e3b00e5afc76fde6c0 [sys.catalog]: This master's current role is: LEADER
I20260620 09:29:01.522135 31983 sys_catalog.cc:458] T 00000000000000000000000000000000 P c18c7503906044e3b00e5afc76fde6c0 [sys.catalog]: This master's current role is: LEADER
I20260620 09:29:01.522455 31990 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:29:01.523039 31990 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:29:01.523061 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 31910
I20260620 09:29:01.523169 16558 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/master-0/wal/instance
I20260620 09:29:01.523361 16558 external_mini_cluster.cc:1428] Setting key ae4ad1ed176d9cf904444b223b55b75b
I20260620 09:29:01.524704 31990 catalog_manager.cc:1347] Generated new cluster ID: 6ef750a9a6cc451da9eb3ed2ee1c6e85
I20260620 09:29:01.524757 31990 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 09:29:01 dist-test-slave-706q krb5kdc[31897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947741, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.43.190@KRBTEST.COM
I20260620 09:29:01.533519 32000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:29:01.524078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33272 (local address 127.16.43.190:44381)
0620 09:29:01.524249 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 09:29:01.524260 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 09:29:01.524299 (+ 39us) server_negotiation.cc:407] Connection header received
0620 09:29:01.524425 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:29:01.524431 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:29:01.524476 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:29:01.524611 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:29:01.525282 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:29:01.526219 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:29:01.527009 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:29:01.527462 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:29:01.527607 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:29:01.530166 (+ 2559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:29:01.530170 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:29:01.530171 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:29:01.530183 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:29:01.531918 (+ 1735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:29:01.532306 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:29:01.532309 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:29:01.532310 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:29:01.532403 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:29:01.532624 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:29:01.532625 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:29:01.532625 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:29:01.532882 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:29:01.533034 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0620 09:29:01.533148 (+ 114us) server_negotiation.cc:299] Negotiation successful
0620 09:29:01.533304 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":82,"threads_started":1}
I20260620 09:29:01.542680 31990 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:29:01.543430 31990 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:29:01.547430 31990 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c18c7503906044e3b00e5afc76fde6c0: Generated new TSK 0
I20260620 09:29:01.547704 31990 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.43.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.43.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.43.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.129.keytab.
Entry for principal kudu/127.16.43.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.129.keytab.
I20260620 09:29:01.578331 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.43.129:0
--local_ip_for_outbound_sockets=127.16.43.129
--webserver_interface=127.16.43.129
--webserver_port=0
--tserver_master_addrs=127.16.43.190:44381
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.129.keytab
--principal=kudu/127.16.43.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.43.148:41053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:29:01.680470 32004 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:29:01.680723 32004 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:29:01.680753 32004 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:29:01.682085 32004 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:29:01.682139 32004 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:29:01.682206 32004 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.43.129
I20260620 09:29:01.684000 32004 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:41053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.129.keytab
--principal=kudu/127.16.43.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.43.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.43.129
--webserver_port=0
--tserver_master_addrs=127.16.43.190:44381
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32004
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.43.129
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:29:01.684273 32004 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:29:01.684544 32004 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:29:01.689941 32010 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:29:01.690004 32012 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:29:01.689941 32009 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:29:01.690186 32004 server_base.cc:1034] running on GCE node
I20260620 09:29:01.690407 32004 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:29:01.690634 32004 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:29:01.691799 32004 hybrid_clock.cc:648] HybridClock initialized: now 1781947741691762 us; error 48 us; skew 500 ppm
Jun 20 09:29:01 dist-test-slave-706q krb5kdc[31897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947741, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:29:01.694681 32004 init.cc:377] Logged in from keytab as kudu/127.16.43.129@KRBTEST.COM (short username kudu)
I20260620 09:29:01.695083 32004 webserver.cc:460] Webserver started at http://127.16.43.129:36881/ using document root <none> and password file <none>
I20260620 09:29:01.695286 32004 fs_manager.cc:359] Metadata directory not provided
I20260620 09:29:01.695327 32004 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:29:01.695430 32004 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:29:01.696537 32004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-0/data/instance:
uuid: "f4e2727ee8d7413a9aa64423a656905a"
format_stamp: "Formatted at 2026-06-20 09:29:01 on dist-test-slave-706q"
server_key: "9727756739f560c432eeb7b0db23a7d7"
server_key_iv: "c8712d4e099cb95692802b4ce558e040"
server_key_version: "encryptionkey@0"
I20260620 09:29:01.696848 32004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-0/wal/instance:
uuid: "f4e2727ee8d7413a9aa64423a656905a"
format_stamp: "Formatted at 2026-06-20 09:29:01 on dist-test-slave-706q"
server_key: "9727756739f560c432eeb7b0db23a7d7"
server_key_iv: "c8712d4e099cb95692802b4ce558e040"
server_key_version: "encryptionkey@0"
I20260620 09:29:01.698135 32004 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 09:29:01.698882 32019 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:29:01.699110 32004 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:29:01.699211 32004 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-0/wal
uuid: "f4e2727ee8d7413a9aa64423a656905a"
format_stamp: "Formatted at 2026-06-20 09:29:01 on dist-test-slave-706q"
server_key: "9727756739f560c432eeb7b0db23a7d7"
server_key_iv: "c8712d4e099cb95692802b4ce558e040"
server_key_version: "encryptionkey@0"
I20260620 09:29:01.699301 32004 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:29:01.725766 32004 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:29:01.728453 32004 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:29:01.728613 32004 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:29:01.728901 32004 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:29:01.729280 32004 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:29:01.729339 32004 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:29:01.729384 32004 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:29:01.729414 32004 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:29:01.735623 32004 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.129:42439
I20260620 09:29:01.735684 32132 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.129:42439 every 8 connection(s)
I20260620 09:29:01.736053 32004 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-0/data/info.pb
Jun 20 09:29:01 dist-test-slave-706q krb5kdc[31897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947741, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.129@KRBTEST.COM for kudu/127.16.43.190@KRBTEST.COM
I20260620 09:29:01.744724 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 32004
I20260620 09:29:01.744871 16558 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-0/wal/instance
I20260620 09:29:01.745075 16558 external_mini_cluster.cc:1428] Setting key bd0d5f4d13df4aee18c49d9af1098dfd
I20260620 09:29:01.746475 32135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:29:01.736811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:44381 (local address 127.16.43.129:40963)
0620 09:29:01.737019 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0620 09:29:01.737072 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 09:29:01.737590 (+ 518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:29:01.737888 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:29:01.737891 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:29:01.738481 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:29:01.738681 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:29:01.738685 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:29:01.739435 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:29:01.739436 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:29:01.740277 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:29:01.740279 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:29:01.740694 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:29:01.740696 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:29:01.740828 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:29:01.741403 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:29:01.741448 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:29:01.743056 (+ 1608us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:29:01.744934 (+ 1878us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:29:01.744943 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:29:01.744945 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:29:01.745235 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:29:01.745482 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:29:01.745484 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:29:01.745484 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:29:01.745610 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:29:01.745931 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:29:01.745937 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:29:01.746067 (+ 130us) client_negotiation.cc:769] Sending connection context
0620 09:29:01.746162 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 09:29:01.746292 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":87,"threads_started":1}
I20260620 09:29:01.746483 32000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:29:01.736892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.129:40963 (local address 127.16.43.190:44381)
0620 09:29:01.736997 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0620 09:29:01.736999 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:29:01.737161 (+ 162us) server_negotiation.cc:407] Connection header received
0620 09:29:01.737700 (+ 539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:29:01.737701 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:29:01.737717 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:29:01.737765 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:29:01.738798 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:29:01.739324 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:29:01.740371 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:29:01.740595 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:29:01.740682 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:29:01.743172 (+ 2490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:29:01.743174 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:29:01.743175 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:29:01.743193 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:29:01.744789 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:29:01.745324 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:29:01.745324 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:29:01.745325 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:29:01.745384 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:29:01.745688 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:29:01.745689 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:29:01.745690 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:29:01.745854 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:29:01.745932 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 09:29:01.746227 (+ 295us) server_negotiation.cc:299] Negotiation successful
0620 09:29:01.746337 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 09:29:01.747017 32133 heartbeater.cc:347] Connected to a master server at 127.16.43.190:44381
I20260620 09:29:01.747128 32133 heartbeater.cc:464] Registering TS with master...
I20260620 09:29:01.747443 32133 heartbeater.cc:511] Master 127.16.43.190:44381 requested a full tablet report, sending...
I20260620 09:29:01.748155 31942 ts_manager.cc:194] Registered new tserver with Master: f4e2727ee8d7413a9aa64423a656905a (127.16.43.129:42439)
I20260620 09:29:01.749116 31942 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.43.129@KRBTEST.COM'} at 127.16.43.129:40963
WARNING: no policy specified for kudu/127.16.43.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.43.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.43.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.130.keytab.
Entry for principal kudu/127.16.43.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.130.keytab.
I20260620 09:29:01.774420 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.43.130:0
--local_ip_for_outbound_sockets=127.16.43.130
--webserver_interface=127.16.43.130
--webserver_port=0
--tserver_master_addrs=127.16.43.190:44381
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.130.keytab
--principal=kudu/127.16.43.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.43.148:41053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:29:01.858054 32138 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:29:01.858279 32138 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:29:01.858304 32138 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:29:01.859591 32138 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:29:01.859643 32138 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:29:01.859671 32138 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.43.130
I20260620 09:29:01.861101 32138 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:41053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.130.keytab
--principal=kudu/127.16.43.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.43.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.43.130
--webserver_port=0
--tserver_master_addrs=127.16.43.190:44381
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32138
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.43.130
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:29:01.861299 32138 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:29:01.861476 32138 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:29:01.866703 32144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:29:01.866722 32143 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:29:01.866680 32146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:29:01.866989 32138 server_base.cc:1034] running on GCE node
I20260620 09:29:01.867269 32138 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:29:01.867502 32138 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:29:01.868786 32138 hybrid_clock.cc:648] HybridClock initialized: now 1781947741868762 us; error 40 us; skew 500 ppm
Jun 20 09:29:01 dist-test-slave-706q krb5kdc[31897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947741, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:29:01.871517 32138 init.cc:377] Logged in from keytab as kudu/127.16.43.130@KRBTEST.COM (short username kudu)
I20260620 09:29:01.872134 32138 webserver.cc:460] Webserver started at http://127.16.43.130:40973/ using document root <none> and password file <none>
I20260620 09:29:01.872407 32138 fs_manager.cc:359] Metadata directory not provided
I20260620 09:29:01.872488 32138 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:29:01.872651 32138 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:29:01.873814 32138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-1/data/instance:
uuid: "c87b67aa6f1a434c85643b5029ea8a91"
format_stamp: "Formatted at 2026-06-20 09:29:01 on dist-test-slave-706q"
server_key: "516ea6a6d812ad1e363f11cdfb7b7c73"
server_key_iv: "ca67562decf88946b3c07737280ab04c"
server_key_version: "encryptionkey@0"
I20260620 09:29:01.874215 32138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-1/wal/instance:
uuid: "c87b67aa6f1a434c85643b5029ea8a91"
format_stamp: "Formatted at 2026-06-20 09:29:01 on dist-test-slave-706q"
server_key: "516ea6a6d812ad1e363f11cdfb7b7c73"
server_key_iv: "ca67562decf88946b3c07737280ab04c"
server_key_version: "encryptionkey@0"
I20260620 09:29:01.875898 32138 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.002s
I20260620 09:29:01.876861 32153 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:29:01.877081 32138 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:29:01.877189 32138 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-1/wal
uuid: "c87b67aa6f1a434c85643b5029ea8a91"
format_stamp: "Formatted at 2026-06-20 09:29:01 on dist-test-slave-706q"
server_key: "516ea6a6d812ad1e363f11cdfb7b7c73"
server_key_iv: "ca67562decf88946b3c07737280ab04c"
server_key_version: "encryptionkey@0"
I20260620 09:29:01.877281 32138 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:29:01.889566 32138 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:29:01.891810 32138 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:29:01.892009 32138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:29:01.892256 32138 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:29:01.892568 32138 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:29:01.892680 32138 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:29:01.892707 32138 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:29:01.892720 32138 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:29:01.898716 32138 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.130:43087
I20260620 09:29:01.898804 32266 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.130:43087 every 8 connection(s)
I20260620 09:29:01.899142 32138 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-1/data/info.pb
I20260620 09:29:01.899547 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 32138
I20260620 09:29:01.899663 16558 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-1/wal/instance
I20260620 09:29:01.899922 16558 external_mini_cluster.cc:1428] Setting key 7b448c8cf23887341c153be7d1515659
Jun 20 09:29:01 dist-test-slave-706q krb5kdc[31897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947741, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.130@KRBTEST.COM for kudu/127.16.43.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.43.131@KRBTEST.COM; defaulting to no policy
I20260620 09:29:01.910111 32000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:29:01.900104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.130:54683 (local address 127.16.43.190:44381)
0620 09:29:01.900162 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 09:29:01.900165 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:29:01.900452 (+ 287us) server_negotiation.cc:407] Connection header received
0620 09:29:01.901102 (+ 650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:29:01.901104 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:29:01.901116 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:29:01.901176 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:29:01.902304 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:29:01.902812 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:29:01.904043 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:29:01.904251 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:29:01.904340 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:29:01.906816 (+ 2476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:29:01.906818 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:29:01.906819 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:29:01.906828 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:29:01.908630 (+ 1802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:29:01.909093 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:29:01.909095 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:29:01.909095 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:29:01.909163 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:29:01.909408 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:29:01.909409 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:29:01.909409 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:29:01.909542 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:29:01.909603 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 09:29:01.909878 (+ 275us) server_negotiation.cc:299] Negotiation successful
0620 09:29:01.909963 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260620 09:29:01.910133 32269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:29:01.900003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:44381 (local address 127.16.43.130:54683)
0620 09:29:01.900302 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0620 09:29:01.900345 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 09:29:01.900984 (+ 639us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:29:01.901241 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:29:01.901245 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:29:01.901884 (+ 639us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:29:01.902208 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:29:01.902214 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:29:01.902912 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:29:01.902913 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:29:01.903953 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:29:01.903955 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:29:01.904350 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:29:01.904351 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:29:01.904483 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:29:01.905027 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:29:01.905036 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:29:01.906706 (+ 1670us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:29:01.908723 (+ 2017us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:29:01.908729 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:29:01.908730 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:29:01.909018 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:29:01.909248 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:29:01.909248 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:29:01.909248 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:29:01.909344 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:29:01.909604 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:29:01.909609 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:29:01.909720 (+ 111us) client_negotiation.cc:769] Sending connection context
0620 09:29:01.909807 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 09:29:01.909927 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":89,"threads_started":1}
I20260620 09:29:01.910730 32267 heartbeater.cc:347] Connected to a master server at 127.16.43.190:44381
I20260620 09:29:01.910830 32267 heartbeater.cc:464] Registering TS with master...
I20260620 09:29:01.911012 32267 heartbeater.cc:511] Master 127.16.43.190:44381 requested a full tablet report, sending...
I20260620 09:29:01.911429 31942 ts_manager.cc:194] Registered new tserver with Master: c87b67aa6f1a434c85643b5029ea8a91 (127.16.43.130:43087)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.43.131@KRBTEST.COM" created.
I20260620 09:29:01.912036 31942 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.43.130@KRBTEST.COM'} at 127.16.43.130:54683
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.43.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.131.keytab.
Entry for principal kudu/127.16.43.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.131.keytab.
I20260620 09:29:01.927107 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.43.131:0
--local_ip_for_outbound_sockets=127.16.43.131
--webserver_interface=127.16.43.131
--webserver_port=0
--tserver_master_addrs=127.16.43.190:44381
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.131.keytab
--principal=kudu/127.16.43.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.43.148:41053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:29:02.009516 32272 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:29:02.009775 32272 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:29:02.009819 32272 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:29:02.010987 32272 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:29:02.011060 32272 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:29:02.011106 32272 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.43.131
I20260620 09:29:02.012669 32272 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:41053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.131.keytab
--principal=kudu/127.16.43.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.43.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.43.131
--webserver_port=0
--tserver_master_addrs=127.16.43.190:44381
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32272
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.43.131
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:29:02.012961 32272 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:29:02.013202 32272 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:29:02.018872 32278 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:29:02.019124 32272 server_base.cc:1034] running on GCE node
W20260620 09:29:02.018848 32277 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:29:02.019515 32280 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:29:02.019757 32272 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:29:02.020087 32272 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:29:02.021268 32272 hybrid_clock.cc:648] HybridClock initialized: now 1781947742021246 us; error 39 us; skew 500 ppm
Jun 20 09:29:02 dist-test-slave-706q krb5kdc[31897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947742, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:29:02.025241 32272 init.cc:377] Logged in from keytab as kudu/127.16.43.131@KRBTEST.COM (short username kudu)
I20260620 09:29:02.025826 32272 webserver.cc:460] Webserver started at http://127.16.43.131:34609/ using document root <none> and password file <none>
I20260620 09:29:02.026269 32272 fs_manager.cc:359] Metadata directory not provided
I20260620 09:29:02.026347 32272 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:29:02.026492 32272 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:29:02.028220 32272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-2/data/instance:
uuid: "804331f690c6404db86d9c664915e9d2"
format_stamp: "Formatted at 2026-06-20 09:29:02 on dist-test-slave-706q"
server_key: "76d950b760a461df80d6a522e8c6077e"
server_key_iv: "29895f931e63037c03a0a64c62b1fda6"
server_key_version: "encryptionkey@0"
I20260620 09:29:02.028645 32272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-2/wal/instance:
uuid: "804331f690c6404db86d9c664915e9d2"
format_stamp: "Formatted at 2026-06-20 09:29:02 on dist-test-slave-706q"
server_key: "76d950b760a461df80d6a522e8c6077e"
server_key_iv: "29895f931e63037c03a0a64c62b1fda6"
server_key_version: "encryptionkey@0"
I20260620 09:29:02.031042 32272 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260620 09:29:02.032159 32287 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:29:02.032316 32272 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 09:29:02.032407 32272 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-2/wal
uuid: "804331f690c6404db86d9c664915e9d2"
format_stamp: "Formatted at 2026-06-20 09:29:02 on dist-test-slave-706q"
server_key: "76d950b760a461df80d6a522e8c6077e"
server_key_iv: "29895f931e63037c03a0a64c62b1fda6"
server_key_version: "encryptionkey@0"
I20260620 09:29:02.032485 32272 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:29:02.053037 32272 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:29:02.056013 32272 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:29:02.056181 32272 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:29:02.056463 32272 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:29:02.056851 32272 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:29:02.056905 32272 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:29:02.056957 32272 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:29:02.056989 32272 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:29:02.065300 32272 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.131:41523
I20260620 09:29:02.065358 32400 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.131:41523 every 8 connection(s)
I20260620 09:29:02.065784 32272 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-2/data/info.pb
Jun 20 09:29:02 dist-test-slave-706q krb5kdc[31897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947742, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.131@KRBTEST.COM for kudu/127.16.43.190@KRBTEST.COM
I20260620 09:29:02.072993 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 32272
I20260620 09:29:02.073097 16558 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-2/wal/instance
I20260620 09:29:02.073302 16558 external_mini_cluster.cc:1428] Setting key 5cf37a9d4a8e4bf5aafc8f08c2ec2d54
I20260620 09:29:02.075582 32000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:29:02.066587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.131:52655 (local address 127.16.43.190:44381)
0620 09:29:02.066646 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0620 09:29:02.066648 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:29:02.066960 (+ 312us) server_negotiation.cc:407] Connection header received
0620 09:29:02.067639 (+ 679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:29:02.067640 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:29:02.067655 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:29:02.067712 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:29:02.068624 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:29:02.069121 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:29:02.070147 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:29:02.070330 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:29:02.070390 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:29:02.072544 (+ 2154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:29:02.072546 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:29:02.072546 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:29:02.072554 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:29:02.074097 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:29:02.074523 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:29:02.074524 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:29:02.074525 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:29:02.074589 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:29:02.074900 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:29:02.074901 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:29:02.074901 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:29:02.075024 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:29:02.075099 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 09:29:02.075376 (+ 277us) server_negotiation.cc:299] Negotiation successful
0620 09:29:02.075460 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 09:29:02.075793 32403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:29:02.066594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:44381 (local address 127.16.43.131:52655)
0620 09:29:02.066798 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0620 09:29:02.066853 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 09:29:02.067474 (+ 621us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:29:02.067776 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:29:02.067779 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:29:02.068346 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:29:02.068541 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:29:02.068546 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:29:02.069207 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:29:02.069208 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:29:02.070069 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:29:02.070071 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:29:02.070410 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:29:02.070411 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:29:02.070505 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:29:02.070988 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:29:02.070997 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:29:02.072444 (+ 1447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:29:02.074205 (+ 1761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:29:02.074212 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:29:02.074214 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:29:02.074442 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:29:02.074659 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:29:02.074660 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:29:02.074660 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:29:02.074819 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:29:02.075090 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:29:02.075097 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:29:02.075239 (+ 142us) client_negotiation.cc:769] Sending connection context
0620 09:29:02.075354 (+ 115us) client_negotiation.cc:240] Negotiation successful
0620 09:29:02.075515 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":109,"threads_started":1}
I20260620 09:29:02.076272 32401 heartbeater.cc:347] Connected to a master server at 127.16.43.190:44381
I20260620 09:29:02.076375 32401 heartbeater.cc:464] Registering TS with master...
I20260620 09:29:02.076555 32401 heartbeater.cc:511] Master 127.16.43.190:44381 requested a full tablet report, sending...
I20260620 09:29:02.076975 31942 ts_manager.cc:194] Registered new tserver with Master: 804331f690c6404db86d9c664915e9d2 (127.16.43.131:41523)
I20260620 09:29:02.077309 31942 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.43.131@KRBTEST.COM'} at 127.16.43.131:52655
I20260620 09:29:02.085038 16558 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 09:29:02.092973 32000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:29:02.086457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33280 (local address 127.16.43.190:44381)
0620 09:29:02.086537 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0620 09:29:02.086539 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:29:02.086672 (+ 133us) server_negotiation.cc:407] Connection header received
0620 09:29:02.086702 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:29:02.086702 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:29:02.086713 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:29:02.086787 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:29:02.087311 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:29:02.087898 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:29:02.088633 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:29:02.088866 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:29:02.088936 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:29:02.090054 (+ 1118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:29:02.090056 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:29:02.090057 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:29:02.090063 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:29:02.091731 (+ 1668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:29:02.092053 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:29:02.092054 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:29:02.092054 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:29:02.092124 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:29:02.092296 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:29:02.092297 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:29:02.092297 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:29:02.092458 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:29:02.092541 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 09:29:02.092729 (+ 188us) server_negotiation.cc:299] Negotiation successful
0620 09:29:02.092831 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 09:29:02.094848 31942 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33280:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 09:29:02.096712 31942 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 09:29:02.103689 32415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:29:02.099389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57206 (local address 127.16.43.131:41523)
0620 09:29:02.099935 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0620 09:29:02.099939 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:29:02.099949 (+ 10us) server_negotiation.cc:407] Connection header received
0620 09:29:02.099985 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:29:02.099987 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:29:02.100004 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:29:02.100081 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:29:02.100279 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:29:02.101316 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:29:02.102680 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:29:02.103216 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:29:02.103309 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:29:02.103382 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 09:29:02.103550 (+ 168us) server_negotiation.cc:299] Negotiation successful
0620 09:29:02.103597 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":510,"thread_start_us":73,"threads_started":1}
I20260620 09:29:02.103679 32412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:29:02.099213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.131:41523 (local address 127.0.0.1:57206)
0620 09:29:02.099507 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0620 09:29:02.099519 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 09:29:02.099609 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:29:02.100164 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:29:02.100165 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:29:02.100167 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:29:02.100204 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:29:02.100206 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:29:02.101449 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:29:02.101450 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:29:02.102596 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:29:02.102598 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:29:02.103307 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:29:02.103308 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:29:02.103386 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:29:02.103401 (+ 15us) client_negotiation.cc:769] Sending connection context
0620 09:29:02.103509 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 09:29:02.103597 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"mutex_wait_us":29,"thread_start_us":93,"threads_started":1}
I20260620 09:29:02.103873 32413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:29:02.099209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.130:43087 (local address 127.0.0.1:56088)
0620 09:29:02.099366 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 09:29:02.099408 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 09:29:02.099479 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:29:02.099868 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:29:02.099874 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:29:02.099877 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:29:02.099953 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:29:02.099955 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:29:02.101286 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:29:02.101287 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:29:02.102390 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:29:02.102393 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:29:02.103452 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:29:02.103454 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:29:02.103562 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:29:02.103566 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 09:29:02.103611 (+ 45us) client_negotiation.cc:240] Negotiation successful
0620 09:29:02.103661 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"mutex_wait_us":27,"thread_start_us":44,"threads_started":1}
I20260620 09:29:02.103871 32414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:29:02.099313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56088 (local address 127.16.43.130:43087)
0620 09:29:02.099611 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0620 09:29:02.099616 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:29:02.099626 (+ 10us) server_negotiation.cc:407] Connection header received
0620 09:29:02.099669 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:29:02.099671 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:29:02.099688 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:29:02.099795 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:29:02.100046 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:29:02.101185 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:29:02.102515 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:29:02.103285 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:29:02.103359 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:29:02.103415 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 09:29:02.103728 (+ 313us) server_negotiation.cc:299] Negotiation successful
0620 09:29:02.103772 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":103,"threads_started":1}
I20260620 09:29:02.103968 32417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:29:02.099650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52230 (local address 127.16.43.129:42439)
0620 09:29:02.099811 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 09:29:02.099816 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:29:02.100020 (+ 204us) server_negotiation.cc:407] Connection header received
0620 09:29:02.100066 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:29:02.100068 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:29:02.100085 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:29:02.100164 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:29:02.100283 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:29:02.100977 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:29:02.102572 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:29:02.103296 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:29:02.103359 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:29:02.103414 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 09:29:02.103709 (+ 295us) server_negotiation.cc:299] Negotiation successful
0620 09:29:02.103880 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":158,"threads_started":1}
I20260620 09:29:02.104118 32416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:29:02.099409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.129:42439 (local address 127.0.0.1:52230)
0620 09:29:02.099903 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0620 09:29:02.099914 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 09:29:02.099964 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:29:02.100159 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:29:02.100160 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:29:02.100162 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:29:02.100214 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:29:02.100216 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:29:02.101085 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:29:02.101086 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:29:02.102498 (+ 1412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:29:02.102501 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:29:02.103454 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:29:02.103454 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:29:02.103550 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:29:02.103555 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 09:29:02.103608 (+ 53us) client_negotiation.cc:240] Negotiation successful
0620 09:29:02.103661 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"mutex_wait_us":66,"thread_start_us":67,"threads_started":1}
I20260620 09:29:02.104637 32201 tablet_service.cc:1467] Processing CreateTablet for tablet 5d97072dd4574ae989acbd1f63be2d07 (DEFAULT_TABLE table=test-table [id=aad942f7ef44461ab8bdf925f59a3116]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:29:02.104803 32067 tablet_service.cc:1467] Processing CreateTablet for tablet 5d97072dd4574ae989acbd1f63be2d07 (DEFAULT_TABLE table=test-table [id=aad942f7ef44461ab8bdf925f59a3116]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:29:02.105041 32201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d97072dd4574ae989acbd1f63be2d07. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:29:02.105052 32335 tablet_service.cc:1467] Processing CreateTablet for tablet 5d97072dd4574ae989acbd1f63be2d07 (DEFAULT_TABLE table=test-table [id=aad942f7ef44461ab8bdf925f59a3116]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:29:02.105151 32067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d97072dd4574ae989acbd1f63be2d07. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:29:02.105336 32335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d97072dd4574ae989acbd1f63be2d07. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:29:02.107924 32418 tablet_bootstrap.cc:492] T 5d97072dd4574ae989acbd1f63be2d07 P f4e2727ee8d7413a9aa64423a656905a: Bootstrap starting.
I20260620 09:29:02.108433 32419 tablet_bootstrap.cc:492] T 5d97072dd4574ae989acbd1f63be2d07 P c87b67aa6f1a434c85643b5029ea8a91: Bootstrap starting.
I20260620 09:29:02.108736 32418 tablet_bootstrap.cc:654] T 5d97072dd4574ae989acbd1f63be2d07 P f4e2727ee8d7413a9aa64423a656905a: Neither blocks nor log segments found. Creating new log.
I20260620 09:29:02.109125 32419 tablet_bootstrap.cc:654] T 5d97072dd4574ae989acbd1f63be2d07 P c87b67aa6f1a434c85643b5029ea8a91: Neither blocks nor log segments found. Creating new log.
I20260620 09:29:02.109405 32418 log.cc:824] T 5d97072dd4574ae989acbd1f63be2d07 P f4e2727ee8d7413a9aa64423a656905a: Log is configured to *not* fsync() on all Append() calls
I20260620 09:29:02.109748 32419 log.cc:824] T 5d97072dd4574ae989acbd1f63be2d07 P c87b67aa6f1a434c85643b5029ea8a91: Log is configured to *not* fsync() on all Append() calls
I20260620 09:29:02.110090 32420 tablet_bootstrap.cc:492] T 5d97072dd4574ae989acbd1f63be2d07 P 804331f690c6404db86d9c664915e9d2: Bootstrap starting.
I20260620 09:29:02.110278 32418 tablet_bootstrap.cc:492] T 5d97072dd4574ae989acbd1f63be2d07 P f4e2727ee8d7413a9aa64423a656905a: No bootstrap required, opened a new log
I20260620 09:29:02.110378 32418 ts_tablet_manager.cc:1397] T 5d97072dd4574ae989acbd1f63be2d07 P f4e2727ee8d7413a9aa64423a656905a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 09:29:02.110638 32419 tablet_bootstrap.cc:492] T 5d97072dd4574ae989acbd1f63be2d07 P c87b67aa6f1a434c85643b5029ea8a91: No bootstrap required, opened a new log
I20260620 09:29:02.110745 32419 ts_tablet_manager.cc:1397] T 5d97072dd4574ae989acbd1f63be2d07 P c87b67aa6f1a434c85643b5029ea8a91: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 09:29:02.110774 32420 tablet_bootstrap.cc:654] T 5d97072dd4574ae989acbd1f63be2d07 P 804331f690c6404db86d9c664915e9d2: Neither blocks nor log segments found. Creating new log.
I20260620 09:29:02.111354 32420 log.cc:824] T 5d97072dd4574ae989acbd1f63be2d07 P 804331f690c6404db86d9c664915e9d2: Log is configured to *not* fsync() on all Append() calls
I20260620 09:29:02.112156 32420 tablet_bootstrap.cc:492] T 5d97072dd4574ae989acbd1f63be2d07 P 804331f690c6404db86d9c664915e9d2: No bootstrap required, opened a new log
I20260620 09:29:02.112227 32420 ts_tablet_manager.cc:1397] T 5d97072dd4574ae989acbd1f63be2d07 P 804331f690c6404db86d9c664915e9d2: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260620 09:29:02.112468 32419 raft_consensus.cc:348] T 5d97072dd4574ae989acbd1f63be2d07 P c87b67aa6f1a434c85643b5029ea8a91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "804331f690c6404db86d9c664915e9d2" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 41523 } } peers { permanent_uuid: "f4e2727ee8d7413a9aa64423a656905a" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 42439 } } peers { permanent_uuid: "c87b67aa6f1a434c85643b5029ea8a91" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 43087 } }
I20260620 09:29:02.112668 32419 raft_consensus.cc:374] T 5d97072dd4574ae989acbd1f63be2d07 P c87b67aa6f1a434c85643b5029ea8a91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:29:02.112707 32419 raft_consensus.cc:729] T 5d97072dd4574ae989acbd1f63be2d07 P c87b67aa6f1a434c85643b5029ea8a91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c87b67aa6f1a434c85643b5029ea8a91, State: Initialized, Role: FOLLOWER
I20260620 09:29:02.112723 32418 raft_consensus.cc:348] T 5d97072dd4574ae989acbd1f63be2d07 P f4e2727ee8d7413a9aa64423a656905a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "804331f690c6404db86d9c664915e9d2" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 41523 } } peers { permanent_uuid: "f4e2727ee8d7413a9aa64423a656905a" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 42439 } } peers { permanent_uuid: "c87b67aa6f1a434c85643b5029ea8a91" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 43087 } }
I20260620 09:29:02.112845 32419 consensus_queue.cc:260] T 5d97072dd4574ae989acbd1f63be2d07 P c87b67aa6f1a434c85643b5029ea8a91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "804331f690c6404db86d9c664915e9d2" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 41523 } } peers { permanent_uuid: "f4e2727ee8d7413a9aa64423a656905a" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 42439 } } peers { permanent_uuid: "c87b67aa6f1a434c85643b5029ea8a91" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 43087 } }
I20260620 09:29:02.112908 32418 raft_consensus.cc:374] T 5d97072dd4574ae989acbd1f63be2d07 P f4e2727ee8d7413a9aa64423a656905a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:29:02.112943 32418 raft_consensus.cc:729] T 5d97072dd4574ae989acbd1f63be2d07 P f4e2727ee8d7413a9aa64423a656905a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4e2727ee8d7413a9aa64423a656905a, State: Initialized, Role: FOLLOWER
I20260620 09:29:02.113082 32418 consensus_queue.cc:260] T 5d97072dd4574ae989acbd1f63be2d07 P f4e2727ee8d7413a9aa64423a656905a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "804331f690c6404db86d9c664915e9d2" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 41523 } } peers { permanent_uuid: "f4e2727ee8d7413a9aa64423a656905a" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 42439 } } peers { permanent_uuid: "c87b67aa6f1a434c85643b5029ea8a91" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 43087 } }
I20260620 09:29:02.113158 32419 ts_tablet_manager.cc:1428] T 5d97072dd4574ae989acbd1f63be2d07 P c87b67aa6f1a434c85643b5029ea8a91: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 09:29:02.113291 32267 heartbeater.cc:503] Master 127.16.43.190:44381 was elected leader, sending a full tablet report...
I20260620 09:29:02.113363 32418 ts_tablet_manager.cc:1428] T 5d97072dd4574ae989acbd1f63be2d07 P f4e2727ee8d7413a9aa64423a656905a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 09:29:02.113474 32133 heartbeater.cc:503] Master 127.16.43.190:44381 was elected leader, sending a full tablet report...
I20260620 09:29:02.114290 32420 raft_consensus.cc:348] T 5d97072dd4574ae989acbd1f63be2d07 P 804331f690c6404db86d9c664915e9d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "804331f690c6404db86d9c664915e9d2" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 41523 } } peers { permanent_uuid: "f4e2727ee8d7413a9aa64423a656905a" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 42439 } } peers { permanent_uuid: "c87b67aa6f1a434c85643b5029ea8a91" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 43087 } }
I20260620 09:29:02.114521 32420 raft_consensus.cc:374] T 5d97072dd4574ae989acbd1f63be2d07 P 804331f690c6404db86d9c664915e9d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:29:02.114580 32420 raft_consensus.cc:729] T 5d97072dd4574ae989acbd1f63be2d07 P 804331f690c6404db86d9c664915e9d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 804331f690c6404db86d9c664915e9d2, State: Initialized, Role: FOLLOWER
I20260620 09:29:02.114717 32420 consensus_queue.cc:260] T 5d97072dd4574ae989acbd1f63be2d07 P 804331f690c6404db86d9c664915e9d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "804331f690c6404db86d9c664915e9d2" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 41523 } } peers { permanent_uuid: "f4e2727ee8d7413a9aa64423a656905a" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 42439 } } peers { permanent_uuid: "c87b67aa6f1a434c85643b5029ea8a91" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 43087 } }
I20260620 09:29:02.115024 32420 ts_tablet_manager.cc:1428] T 5d97072dd4574ae989acbd1f63be2d07 P 804331f690c6404db86d9c664915e9d2: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260620 09:29:02.115034 32401 heartbeater.cc:503] Master 127.16.43.190:44381 was elected leader, sending a full tablet report...
W20260620 09:29:02.149845 32268 tablet.cc:2369] T 5d97072dd4574ae989acbd1f63be2d07 P c87b67aa6f1a434c85643b5029ea8a91: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 09:29:02.236901 32134 tablet.cc:2369] T 5d97072dd4574ae989acbd1f63be2d07 P f4e2727ee8d7413a9aa64423a656905a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 09:29:02.316460 32402 tablet.cc:2369] T 5d97072dd4574ae989acbd1f63be2d07 P 804331f690c6404db86d9c664915e9d2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 09:29:02.323812 32424 raft_consensus.cc:482] T 5d97072dd4574ae989acbd1f63be2d07 P c87b67aa6f1a434c85643b5029ea8a91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:29:02.323983 32424 raft_consensus.cc:504] T 5d97072dd4574ae989acbd1f63be2d07 P c87b67aa6f1a434c85643b5029ea8a91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "804331f690c6404db86d9c664915e9d2" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 41523 } } peers { permanent_uuid: "f4e2727ee8d7413a9aa64423a656905a" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 42439 } } peers { permanent_uuid: "c87b67aa6f1a434c85643b5029ea8a91" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 43087 } }
I20260620 09:29:02.324374 32424 leader_election.cc:288] T 5d97072dd4574ae989acbd1f63be2d07 P c87b67aa6f1a434c85643b5029ea8a91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 804331f690c6404db86d9c664915e9d2 (127.16.43.131:41523), f4e2727ee8d7413a9aa64423a656905a (127.16.43.129:42439)
I20260620 09:29:02.327803 32269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:29:02.324611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.131:41523 (local address 127.16.43.130:56473)
0620 09:29:02.324675 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0620 09:29:02.324691 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 09:29:02.324770 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:29:02.325010 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:29:02.325011 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:29:02.325013 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:29:02.325091 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:29:02.325094 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:29:02.326115 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:29:02.326116 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:29:02.326906 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:29:02.326909 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:29:02.327508 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:29:02.327509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:29:02.327613 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:29:02.327616 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 09:29:02.327686 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 09:29:02.327725 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260620 09:29:02.327805 32415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:29:02.324800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.130:56473 (local address 127.16.43.131:41523)
0620 09:29:02.324861 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 09:29:02.324863 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:29:02.324874 (+ 11us) server_negotiation.cc:407] Connection header received
0620 09:29:02.324908 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:29:02.324909 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:29:02.324921 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:29:02.324990 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:29:02.325170 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:29:02.326023 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:29:02.327008 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:29:02.327411 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:29:02.327511 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:29:02.327521 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 09:29:02.327691 (+ 170us) server_negotiation.cc:299] Negotiation successful
0620 09:29:02.327733 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 09:29:02.328104 32427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:29:02.324760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.129:42439 (local address 127.16.43.130:43007)
0620 09:29:02.324943 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0620 09:29:02.324952 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 09:29:02.324994 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:29:02.325113 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:29:02.325113 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:29:02.325115 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:29:02.325176 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:29:02.325177 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:29:02.326153 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:29:02.326154 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:29:02.327051 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:29:02.327053 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:29:02.327741 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:29:02.327742 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:29:02.327835 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:29:02.327839 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 09:29:02.327962 (+ 123us) client_negotiation.cc:240] Negotiation successful
0620 09:29:02.328019 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":117,"threads_started":1}
I20260620 09:29:02.328152 32417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:29:02.324907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.130:43007 (local address 127.16.43.129:42439)
0620 09:29:02.324991 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0620 09:29:02.324993 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:29:02.325003 (+ 10us) server_negotiation.cc:407] Connection header received
0620 09:29:02.325042 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:29:02.325042 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:29:02.325055 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:29:02.325101 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:29:02.325256 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:29:02.326022 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:29:02.327154 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:29:02.327649 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:29:02.327740 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:29:02.327750 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 09:29:02.328042 (+ 292us) server_negotiation.cc:299] Negotiation successful
0620 09:29:02.328091 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 09:29:02.328155 32355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d97072dd4574ae989acbd1f63be2d07" candidate_uuid: "c87b67aa6f1a434c85643b5029ea8a91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "804331f690c6404db86d9c664915e9d2" is_pre_election: true
I20260620 09:29:02.328320 32355 raft_consensus.cc:2454] T 5d97072dd4574ae989acbd1f63be2d07 P 804331f690c6404db86d9c664915e9d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c87b67aa6f1a434c85643b5029ea8a91 in term 0.
I20260620 09:29:02.328554 32155 leader_election.cc:302] T 5d97072dd4574ae989acbd1f63be2d07 P c87b67aa6f1a434c85643b5029ea8a91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 804331f690c6404db86d9c664915e9d2, c87b67aa6f1a434c85643b5029ea8a91; no voters:
I20260620 09:29:02.328718 32424 raft_consensus.cc:2781] T 5d97072dd4574ae989acbd1f63be2d07 P c87b67aa6f1a434c85643b5029ea8a91 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 09:29:02.328701 32087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d97072dd4574ae989acbd1f63be2d07" candidate_uuid: "c87b67aa6f1a434c85643b5029ea8a91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4e2727ee8d7413a9aa64423a656905a" is_pre_election: true
I20260620 09:29:02.328788 32424 raft_consensus.cc:482] T 5d97072dd4574ae989acbd1f63be2d07 P c87b67aa6f1a434c85643b5029ea8a91 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 09:29:02.328806 32424 raft_consensus.cc:3037] T 5d97072dd4574ae989acbd1f63be2d07 P c87b67aa6f1a434c85643b5029ea8a91 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:29:02.328861 32087 raft_consensus.cc:2454] T 5d97072dd4574ae989acbd1f63be2d07 P f4e2727ee8d7413a9aa64423a656905a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c87b67aa6f1a434c85643b5029ea8a91 in term 0.
I20260620 09:29:02.329607 32424 raft_consensus.cc:504] T 5d97072dd4574ae989acbd1f63be2d07 P c87b67aa6f1a434c85643b5029ea8a91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "804331f690c6404db86d9c664915e9d2" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 41523 } } peers { permanent_uuid: "f4e2727ee8d7413a9aa64423a656905a" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 42439 } } peers { permanent_uuid: "c87b67aa6f1a434c85643b5029ea8a91" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 43087 } }
I20260620 09:29:02.329741 32424 leader_election.cc:288] T 5d97072dd4574ae989acbd1f63be2d07 P c87b67aa6f1a434c85643b5029ea8a91 [CANDIDATE]: Term 1 election: Requested vote from peers 804331f690c6404db86d9c664915e9d2 (127.16.43.131:41523), f4e2727ee8d7413a9aa64423a656905a (127.16.43.129:42439)
I20260620 09:29:02.329954 32355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d97072dd4574ae989acbd1f63be2d07" candidate_uuid: "c87b67aa6f1a434c85643b5029ea8a91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "804331f690c6404db86d9c664915e9d2"
I20260620 09:29:02.329934 32087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d97072dd4574ae989acbd1f63be2d07" candidate_uuid: "c87b67aa6f1a434c85643b5029ea8a91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4e2727ee8d7413a9aa64423a656905a"
I20260620 09:29:02.330034 32355 raft_consensus.cc:3037] T 5d97072dd4574ae989acbd1f63be2d07 P 804331f690c6404db86d9c664915e9d2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:29:02.330058 32087 raft_consensus.cc:3037] T 5d97072dd4574ae989acbd1f63be2d07 P f4e2727ee8d7413a9aa64423a656905a [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:29:02.330724 32355 raft_consensus.cc:2454] T 5d97072dd4574ae989acbd1f63be2d07 P 804331f690c6404db86d9c664915e9d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c87b67aa6f1a434c85643b5029ea8a91 in term 1.
I20260620 09:29:02.330893 32155 leader_election.cc:302] T 5d97072dd4574ae989acbd1f63be2d07 P c87b67aa6f1a434c85643b5029ea8a91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 804331f690c6404db86d9c664915e9d2, c87b67aa6f1a434c85643b5029ea8a91; no voters:
I20260620 09:29:02.330906 32087 raft_consensus.cc:2454] T 5d97072dd4574ae989acbd1f63be2d07 P f4e2727ee8d7413a9aa64423a656905a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c87b67aa6f1a434c85643b5029ea8a91 in term 1.
I20260620 09:29:02.330994 32424 raft_consensus.cc:2781] T 5d97072dd4574ae989acbd1f63be2d07 P c87b67aa6f1a434c85643b5029ea8a91 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:29:02.331147 32424 raft_consensus.cc:686] T 5d97072dd4574ae989acbd1f63be2d07 P c87b67aa6f1a434c85643b5029ea8a91 [term 1 LEADER]: Becoming Leader. State: Replica: c87b67aa6f1a434c85643b5029ea8a91, State: Running, Role: LEADER
I20260620 09:29:02.331288 32424 consensus_queue.cc:237] T 5d97072dd4574ae989acbd1f63be2d07 P c87b67aa6f1a434c85643b5029ea8a91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "804331f690c6404db86d9c664915e9d2" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 41523 } } peers { permanent_uuid: "f4e2727ee8d7413a9aa64423a656905a" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 42439 } } peers { permanent_uuid: "c87b67aa6f1a434c85643b5029ea8a91" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 43087 } }
I20260620 09:29:02.332165 31941 catalog_manager.cc:5495] T 5d97072dd4574ae989acbd1f63be2d07 P c87b67aa6f1a434c85643b5029ea8a91 reported cstate change: term changed from 0 to 1, leader changed from <none> to c87b67aa6f1a434c85643b5029ea8a91 (127.16.43.130). New cstate: current_term: 1 leader_uuid: "c87b67aa6f1a434c85643b5029ea8a91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "804331f690c6404db86d9c664915e9d2" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 41523 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f4e2727ee8d7413a9aa64423a656905a" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 42439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c87b67aa6f1a434c85643b5029ea8a91" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 43087 } health_report { overall_health: HEALTHY } } }
I20260620 09:29:02.387476 32414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:29:02.384998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56102 (local address 127.16.43.130:43087)
0620 09:29:02.385075 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0620 09:29:02.385077 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:29:02.385085 (+ 8us) server_negotiation.cc:407] Connection header received
0620 09:29:02.385121 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:29:02.385122 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:29:02.385140 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:29:02.385205 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:29:02.385384 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:29:02.385941 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:29:02.386668 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:29:02.386892 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:29:02.386946 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:29:02.387126 (+ 180us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:29:02.387232 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:29:02.387294 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 09:29:02.387332 (+ 38us) server_negotiation.cc:299] Negotiation successful
0620 09:29:02.387352 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 09:29:02.389436 32355 raft_consensus.cc:1261] T 5d97072dd4574ae989acbd1f63be2d07 P 804331f690c6404db86d9c664915e9d2 [term 1 FOLLOWER]: Refusing update from remote peer c87b67aa6f1a434c85643b5029ea8a91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:29:02.389429 32087 raft_consensus.cc:1261] T 5d97072dd4574ae989acbd1f63be2d07 P f4e2727ee8d7413a9aa64423a656905a [term 1 FOLLOWER]: Refusing update from remote peer c87b67aa6f1a434c85643b5029ea8a91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:29:02.389840 32424 consensus_queue.cc:1035] T 5d97072dd4574ae989acbd1f63be2d07 P c87b67aa6f1a434c85643b5029ea8a91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4e2727ee8d7413a9aa64423a656905a" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 42439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:29:02.390000 32428 consensus_queue.cc:1035] T 5d97072dd4574ae989acbd1f63be2d07 P c87b67aa6f1a434c85643b5029ea8a91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "804331f690c6404db86d9c664915e9d2" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 41523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:29:02.396349 32431 mvcc.cc:205] Tried to move back new op lower bound from 7298857952824360960 to 7298857952589471744. Current Snapshot: MvccSnapshot[applied={T|T < 7298857952824360960}]
I20260620 09:29:02.400079 31942 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33280:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:29:02.400168 31942 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33280:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:29:02.400967 31942 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c18c7503906044e3b00e5afc76fde6c0: Sending DeleteTablet for 3 replicas of tablet 5d97072dd4574ae989acbd1f63be2d07
I20260620 09:29:02.401293 32335 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d97072dd4574ae989acbd1f63be2d07 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:29:02 UTC) from {username='kudu', principal='kudu/127.16.43.190@KRBTEST.COM'} at 127.0.0.1:57206
I20260620 09:29:02.401347 32067 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d97072dd4574ae989acbd1f63be2d07 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:29:02 UTC) from {username='kudu', principal='kudu/127.16.43.190@KRBTEST.COM'} at 127.0.0.1:52230
I20260620 09:29:02.401366 32201 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d97072dd4574ae989acbd1f63be2d07 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:29:02 UTC) from {username='kudu', principal='kudu/127.16.43.190@KRBTEST.COM'} at 127.0.0.1:56088
I20260620 09:29:02.401405 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 32004
I20260620 09:29:02.402065 32441 tablet_replica.cc:330] T 5d97072dd4574ae989acbd1f63be2d07 P 804331f690c6404db86d9c664915e9d2: stopping tablet replica
I20260620 09:29:02.402195 32441 raft_consensus.cc:2229] T 5d97072dd4574ae989acbd1f63be2d07 P 804331f690c6404db86d9c664915e9d2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:29:02.402326 32441 raft_consensus.cc:2258] T 5d97072dd4574ae989acbd1f63be2d07 P 804331f690c6404db86d9c664915e9d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:29:02.402747 32441 ts_tablet_manager.cc:1905] T 5d97072dd4574ae989acbd1f63be2d07 P 804331f690c6404db86d9c664915e9d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:29:02.404897 32441 ts_tablet_manager.cc:1918] T 5d97072dd4574ae989acbd1f63be2d07 P 804331f690c6404db86d9c664915e9d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:29:02.404980 32441 log.cc:1196] T 5d97072dd4574ae989acbd1f63be2d07 P 804331f690c6404db86d9c664915e9d2: Deleting WAL directory at /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-2/wal/wals/5d97072dd4574ae989acbd1f63be2d07
I20260620 09:29:02.405248 32441 ts_tablet_manager.cc:1939] T 5d97072dd4574ae989acbd1f63be2d07 P 804331f690c6404db86d9c664915e9d2: Deleting consensus metadata
I20260620 09:29:02.405727 31927 catalog_manager.cc:4841] TS 804331f690c6404db86d9c664915e9d2 (127.16.43.131:41523): tablet 5d97072dd4574ae989acbd1f63be2d07 (table test-table [id=aad942f7ef44461ab8bdf925f59a3116]) successfully deleted
W20260620 09:29:02.407362 31927 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.43.129:42439 (error 108)
I20260620 09:29:02.407514 32442 tablet_replica.cc:330] T 5d97072dd4574ae989acbd1f63be2d07 P c87b67aa6f1a434c85643b5029ea8a91: stopping tablet replica
I20260620 09:29:02.407648 32442 raft_consensus.cc:2229] T 5d97072dd4574ae989acbd1f63be2d07 P c87b67aa6f1a434c85643b5029ea8a91 [term 1 LEADER]: Raft consensus shutting down.
I20260620 09:29:02.407815 32442 raft_consensus.cc:2258] T 5d97072dd4574ae989acbd1f63be2d07 P c87b67aa6f1a434c85643b5029ea8a91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:29:02.408020 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 32138
I20260620 09:29:02.408033 32413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:29:02.407883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.129:42439 (local address 127.0.0.1:52246)
0620 09:29:02.407952 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0620 09:29:02.407997 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.129:42439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260620 09:29:02.408630 31927 catalog_manager.cc:4584] TS f4e2727ee8d7413a9aa64423a656905a (127.16.43.129:42439): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5d97072dd4574ae989acbd1f63be2d07: Network error: Client connection negotiation failed: client connection to 127.16.43.129:42439: connect: Connection refused (error 111)
I20260620 09:29:02.413201 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 32272
I20260620 09:29:02.413326 32413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:29:02.413175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.130:43087 (local address 127.0.0.1:56118)
0620 09:29:02.413237 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0620 09:29:02.413275 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.130:43087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260620 09:29:02.418395 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 31910
I20260620 09:29:22.424120 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.43.190:44381
--webserver_interface=127.16.43.190
--webserver_port=46375
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.190.keytab
--principal=kudu/127.16.43.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.43.148:41053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.43.190:44381
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:29:22.509841 32444 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:29:22.510082 32444 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:29:22.510110 32444 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:29:22.511373 32444 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 09:29:22.511425 32444 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 09:29:22.511439 32444 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 09:29:22.511451 32444 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:29:22.511497 32444 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:29:22.511572 32444 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:29:22.511592 32444 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:29:22.511626 32444 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:29:22.513257 32444 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:41053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.16.43.190:44381
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.190.keytab
--principal=kudu/127.16.43.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.43.190:44381
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.43.190
--webserver_port=46375
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32444
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:29:22.513540 32444 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:29:22.513819 32444 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:29:22.524001 32453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:29:22.524001 32450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:29:22.524001 32451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:29:22.524536 32444 server_base.cc:1034] running on GCE node
I20260620 09:29:22.524804 32444 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:29:22.525161 32444 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:29:22.526316 32444 hybrid_clock.cc:648] HybridClock initialized: now 1781947762526287 us; error 47 us; skew 500 ppm
Jun 20 09:29:22 dist-test-slave-706q krb5kdc[31897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947762, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:29:22.529438 32444 init.cc:377] Logged in from keytab as kudu/127.16.43.190@KRBTEST.COM (short username kudu)
I20260620 09:29:22.530097 32444 webserver.cc:460] Webserver started at http://127.16.43.190:46375/ using document root <none> and password file <none>
I20260620 09:29:22.530373 32444 fs_manager.cc:359] Metadata directory not provided
I20260620 09:29:22.530467 32444 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:29:22.532059 32444 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 09:29:22.532841 32460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:29:22.533090 32444 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:29:22.533215 32444 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/master-0/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/master-0/wal
uuid: "c18c7503906044e3b00e5afc76fde6c0"
format_stamp: "Formatted at 2026-06-20 09:29:01 on dist-test-slave-706q"
server_key: "8460fbc73d47b6d32e6e6108117f9d71"
server_key_iv: "bba2020cb7145bbd5a4d8e9b74f35dab"
server_key_version: "encryptionkey@0"
I20260620 09:29:22.533658 32444 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:29:22.555362 32444 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:29:22.557618 32444 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:29:22.557821 32444 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:29:22.562652 32444 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.190:44381
I20260620 09:29:22.562716 32512 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.190:44381 every 8 connection(s)
I20260620 09:29:22.563159 32444 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
I20260620 09:29:22.565018 32513 sys_catalog.cc:263] Verifying existing consensus state
I20260620 09:29:22.565727 32513 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c18c7503906044e3b00e5afc76fde6c0: Bootstrap starting.
I20260620 09:29:22.570814 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 32444
I20260620 09:29:22.571280 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.43.129:42439
--local_ip_for_outbound_sockets=127.16.43.129
--tserver_master_addrs=127.16.43.190:44381
--webserver_port=36881
--webserver_interface=127.16.43.129
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.129.keytab
--principal=kudu/127.16.43.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.43.148:41053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 09:29:22.577003 32513 log.cc:824] T 00000000000000000000000000000000 P c18c7503906044e3b00e5afc76fde6c0: Log is configured to *not* fsync() on all Append() calls
I20260620 09:29:22.579707 32513 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c18c7503906044e3b00e5afc76fde6c0: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 09:29:22.580089 32513 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c18c7503906044e3b00e5afc76fde6c0: Bootstrap complete.
I20260620 09:29:22.582726 32513 raft_consensus.cc:348] T 00000000000000000000000000000000 P c18c7503906044e3b00e5afc76fde6c0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c18c7503906044e3b00e5afc76fde6c0" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 44381 } }
I20260620 09:29:22.582988 32513 raft_consensus.cc:729] T 00000000000000000000000000000000 P c18c7503906044e3b00e5afc76fde6c0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c18c7503906044e3b00e5afc76fde6c0, State: Initialized, Role: FOLLOWER
I20260620 09:29:22.583184 32513 consensus_queue.cc:260] T 00000000000000000000000000000000 P c18c7503906044e3b00e5afc76fde6c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c18c7503906044e3b00e5afc76fde6c0" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 44381 } }
I20260620 09:29:22.583282 32513 raft_consensus.cc:388] T 00000000000000000000000000000000 P c18c7503906044e3b00e5afc76fde6c0 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:29:22.583336 32513 raft_consensus.cc:482] T 00000000000000000000000000000000 P c18c7503906044e3b00e5afc76fde6c0 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:29:22.583416 32513 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c18c7503906044e3b00e5afc76fde6c0 [term 1 FOLLOWER]: Advancing to term 2
I20260620 09:29:22.584721 32513 raft_consensus.cc:504] T 00000000000000000000000000000000 P c18c7503906044e3b00e5afc76fde6c0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c18c7503906044e3b00e5afc76fde6c0" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 44381 } }
I20260620 09:29:22.584859 32513 leader_election.cc:302] T 00000000000000000000000000000000 P c18c7503906044e3b00e5afc76fde6c0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c18c7503906044e3b00e5afc76fde6c0; no voters:
I20260620 09:29:22.585148 32513 leader_election.cc:288] T 00000000000000000000000000000000 P c18c7503906044e3b00e5afc76fde6c0 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 09:29:22.585187 32517 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c18c7503906044e3b00e5afc76fde6c0 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 09:29:22.585330 32517 raft_consensus.cc:686] T 00000000000000000000000000000000 P c18c7503906044e3b00e5afc76fde6c0 [term 2 LEADER]: Becoming Leader. State: Replica: c18c7503906044e3b00e5afc76fde6c0, State: Running, Role: LEADER
I20260620 09:29:22.585443 32517 consensus_queue.cc:237] T 00000000000000000000000000000000 P c18c7503906044e3b00e5afc76fde6c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c18c7503906044e3b00e5afc76fde6c0" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 44381 } }
I20260620 09:29:22.585536 32513 sys_catalog.cc:564] T 00000000000000000000000000000000 P c18c7503906044e3b00e5afc76fde6c0 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:29:22.585772 32518 sys_catalog.cc:455] T 00000000000000000000000000000000 P c18c7503906044e3b00e5afc76fde6c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "c18c7503906044e3b00e5afc76fde6c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c18c7503906044e3b00e5afc76fde6c0" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 44381 } } }
I20260620 09:29:22.585896 32518 sys_catalog.cc:458] T 00000000000000000000000000000000 P c18c7503906044e3b00e5afc76fde6c0 [sys.catalog]: This master's current role is: LEADER
I20260620 09:29:22.585764 32517 sys_catalog.cc:455] T 00000000000000000000000000000000 P c18c7503906044e3b00e5afc76fde6c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c18c7503906044e3b00e5afc76fde6c0. Latest consensus state: current_term: 2 leader_uuid: "c18c7503906044e3b00e5afc76fde6c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c18c7503906044e3b00e5afc76fde6c0" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 44381 } } }
I20260620 09:29:22.586217 32517 sys_catalog.cc:458] T 00000000000000000000000000000000 P c18c7503906044e3b00e5afc76fde6c0 [sys.catalog]: This master's current role is: LEADER
I20260620 09:29:22.586360 32521 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:29:22.587436 32521 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:29:22.587631 32521 catalog_manager.cc:1259] Loaded cluster ID: 6ef750a9a6cc451da9eb3ed2ee1c6e85
I20260620 09:29:22.587922 32521 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:29:22.588835 32521 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:29:22.589058 32521 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c18c7503906044e3b00e5afc76fde6c0: Loaded TSK: 0
I20260620 09:29:22.596361 32521 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c18c7503906044e3b00e5afc76fde6c0: Generated new TSK 1
I20260620 09:29:22.596487 32521 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 09:29:22.674470 32515 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:29:22.674782 32515 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:29:22.674827 32515 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:29:22.676185 32515 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:29:22.676266 32515 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:29:22.676318 32515 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.43.129
I20260620 09:29:22.678112 32515 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:41053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.129.keytab
--principal=kudu/127.16.43.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.43.129:42439
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.43.129
--webserver_port=36881
--tserver_master_addrs=127.16.43.190:44381
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32515
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.43.129
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:29:22.678426 32515 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:29:22.678704 32515 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:29:22.685104 32543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:29:22.685105 32540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:29:22.685473 32515 server_base.cc:1034] running on GCE node
W20260620 09:29:22.685142 32541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:29:22.685927 32515 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:29:22.686388 32515 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:29:22.687608 32515 hybrid_clock.cc:648] HybridClock initialized: now 1781947762687579 us; error 48 us; skew 500 ppm
Jun 20 09:29:22 dist-test-slave-706q krb5kdc[31897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947762, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:29:22.691298 32515 init.cc:377] Logged in from keytab as kudu/127.16.43.129@KRBTEST.COM (short username kudu)
I20260620 09:29:22.691965 32515 webserver.cc:460] Webserver started at http://127.16.43.129:36881/ using document root <none> and password file <none>
I20260620 09:29:22.692301 32515 fs_manager.cc:359] Metadata directory not provided
I20260620 09:29:22.692394 32515 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:29:22.694200 32515 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 09:29:22.695122 32550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:29:22.695377 32515 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 09:29:22.695515 32515 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-0/wal
uuid: "f4e2727ee8d7413a9aa64423a656905a"
format_stamp: "Formatted at 2026-06-20 09:29:01 on dist-test-slave-706q"
server_key: "9727756739f560c432eeb7b0db23a7d7"
server_key_iv: "c8712d4e099cb95692802b4ce558e040"
server_key_version: "encryptionkey@0"
I20260620 09:29:22.696022 32515 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:29:22.710873 32515 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:29:22.713591 32515 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:29:22.713780 32515 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:29:22.714069 32515 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:29:22.714663 32557 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 09:29:22.716974 32515 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 09:29:22.717049 32515 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.003s user 0.000s sys 0.000s
I20260620 09:29:22.717105 32515 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 09:29:22.717936 32515 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 09:29:22.717995 32515 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260620 09:29:22.718116 32557 tablet_bootstrap.cc:492] T 5d97072dd4574ae989acbd1f63be2d07 P f4e2727ee8d7413a9aa64423a656905a: Bootstrap starting.
I20260620 09:29:22.726478 32515 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.129:42439
I20260620 09:29:22.726934 32515 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-0/data/info.pb
I20260620 09:29:22.728610 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 32515
I20260620 09:29:22.728947 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.43.130:43087
--local_ip_for_outbound_sockets=127.16.43.130
--tserver_master_addrs=127.16.43.190:44381
--webserver_port=40973
--webserver_interface=127.16.43.130
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.130.keytab
--principal=kudu/127.16.43.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.43.148:41053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 09:29:22.731006 32557 log.cc:824] T 5d97072dd4574ae989acbd1f63be2d07 P f4e2727ee8d7413a9aa64423a656905a: Log is configured to *not* fsync() on all Append() calls
I20260620 09:29:22.731916 32664 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.129:42439 every 8 connection(s)
I20260620 09:29:22.738269 32557 tablet_bootstrap.cc:492] T 5d97072dd4574ae989acbd1f63be2d07 P f4e2727ee8d7413a9aa64423a656905a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 09:29:22.738651 32557 tablet_bootstrap.cc:492] T 5d97072dd4574ae989acbd1f63be2d07 P f4e2727ee8d7413a9aa64423a656905a: Bootstrap complete.
I20260620 09:29:22.739794 32557 ts_tablet_manager.cc:1397] T 5d97072dd4574ae989acbd1f63be2d07 P f4e2727ee8d7413a9aa64423a656905a: Time spent bootstrapping tablet: real 0.022s user 0.004s sys 0.013s
I20260620 09:29:22.742457 32557 raft_consensus.cc:348] T 5d97072dd4574ae989acbd1f63be2d07 P f4e2727ee8d7413a9aa64423a656905a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "804331f690c6404db86d9c664915e9d2" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 41523 } } peers { permanent_uuid: "f4e2727ee8d7413a9aa64423a656905a" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 42439 } } peers { permanent_uuid: "c87b67aa6f1a434c85643b5029ea8a91" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 43087 } }
I20260620 09:29:22.742825 32557 raft_consensus.cc:729] T 5d97072dd4574ae989acbd1f63be2d07 P f4e2727ee8d7413a9aa64423a656905a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4e2727ee8d7413a9aa64423a656905a, State: Initialized, Role: FOLLOWER
I20260620 09:29:22.743000 32557 consensus_queue.cc:260] T 5d97072dd4574ae989acbd1f63be2d07 P f4e2727ee8d7413a9aa64423a656905a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "804331f690c6404db86d9c664915e9d2" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 41523 } } peers { permanent_uuid: "f4e2727ee8d7413a9aa64423a656905a" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 42439 } } peers { permanent_uuid: "c87b67aa6f1a434c85643b5029ea8a91" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 43087 } }
I20260620 09:29:22.743383 32557 ts_tablet_manager.cc:1428] T 5d97072dd4574ae989acbd1f63be2d07 P f4e2727ee8d7413a9aa64423a656905a: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
Jun 20 09:29:22 dist-test-slave-706q krb5kdc[31897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947762, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.129@KRBTEST.COM for kudu/127.16.43.190@KRBTEST.COM
I20260620 09:29:22.754735 32668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:29:22.727981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.129:44819 (local address 127.16.43.190:44381)
0620 09:29:22.728247 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0620 09:29:22.728257 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 09:29:22.740047 (+ 11790us) server_negotiation.cc:407] Connection header received
0620 09:29:22.741277 (+ 1230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:29:22.741298 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:29:22.741385 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:29:22.741606 (+ 221us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:29:22.743225 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:29:22.744381 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:29:22.746012 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:29:22.746768 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:29:22.746862 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:29:22.750236 (+ 3374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:29:22.750241 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:29:22.750242 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:29:22.750258 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:29:22.752365 (+ 2107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:29:22.753082 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:29:22.753086 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:29:22.753086 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:29:22.753196 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:29:22.753547 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:29:22.753548 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:29:22.753549 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:29:22.753850 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:29:22.753932 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 09:29:22.754381 (+ 449us) server_negotiation.cc:299] Negotiation successful
0620 09:29:22.754528 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":131,"threads_started":1}
I20260620 09:29:22.754827 32667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:29:22.727851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:44381 (local address 127.16.43.129:44819)
0620 09:29:22.739727 (+ 11876us) negotiation.cc:107] Waiting for socket to connect
0620 09:29:22.739830 (+ 103us) client_negotiation.cc:174] Beginning negotiation
0620 09:29:22.740929 (+ 1099us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:29:22.741898 (+ 969us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:29:22.741904 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:29:22.742354 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:29:22.742691 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:29:22.742699 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:29:22.744593 (+ 1894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:29:22.744594 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:29:22.745876 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:29:22.745881 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:29:22.747026 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:29:22.747028 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:29:22.747159 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:29:22.748004 (+ 845us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:29:22.748016 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:29:22.750053 (+ 2037us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:29:22.752493 (+ 2440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:29:22.752506 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:29:22.752510 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:29:22.752934 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:29:22.753284 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:29:22.753285 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:29:22.753285 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:29:22.753452 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:29:22.753935 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:29:22.753942 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:29:22.754164 (+ 222us) client_negotiation.cc:769] Sending connection context
0620 09:29:22.754306 (+ 142us) client_negotiation.cc:240] Negotiation successful
0620 09:29:22.754543 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11770,"thread_start_us":140,"threads_started":1}
I20260620 09:29:22.755964 32665 heartbeater.cc:347] Connected to a master server at 127.16.43.190:44381
I20260620 09:29:22.756099 32665 heartbeater.cc:464] Registering TS with master...
I20260620 09:29:22.756357 32665 heartbeater.cc:511] Master 127.16.43.190:44381 requested a full tablet report, sending...
I20260620 09:29:22.757359 32477 ts_manager.cc:194] Registered new tserver with Master: f4e2727ee8d7413a9aa64423a656905a (127.16.43.129:42439)
I20260620 09:29:22.758664 32477 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.43.129@KRBTEST.COM'} at 127.16.43.129:44819
I20260620 09:29:22.760046 32665 heartbeater.cc:503] Master 127.16.43.190:44381 was elected leader, sending a full tablet report...
Jun 20 09:29:22 dist-test-slave-706q krb5kdc[31897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947762, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.190@KRBTEST.COM for kudu/127.16.43.129@KRBTEST.COM
I20260620 09:29:22.769872 32673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:29:22.758243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51494 (local address 127.16.43.129:42439)
0620 09:29:22.758442 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 09:29:22.758447 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:29:22.759302 (+ 855us) server_negotiation.cc:407] Connection header received
0620 09:29:22.759451 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:29:22.759454 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:29:22.759475 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:29:22.759627 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:29:22.760223 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:29:22.761210 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:29:22.762633 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:29:22.763068 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:29:22.763143 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:29:22.766068 (+ 2925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:29:22.766078 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:29:22.766078 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:29:22.766087 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:29:22.768187 (+ 2100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:29:22.768826 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:29:22.768830 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:29:22.768830 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:29:22.768903 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:29:22.769155 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:29:22.769155 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:29:22.769156 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:29:22.769322 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:29:22.769401 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 09:29:22.769647 (+ 246us) server_negotiation.cc:299] Negotiation successful
0620 09:29:22.769731 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":94,"threads_started":1}
I20260620 09:29:22.769899 32672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:29:22.758115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.129:42439 (local address 127.0.0.1:51494)
0620 09:29:22.759162 (+ 1047us) negotiation.cc:107] Waiting for socket to connect
0620 09:29:22.759212 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 09:29:22.759320 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:29:22.759627 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:29:22.759634 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:29:22.760027 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:29:22.760094 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:29:22.760097 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:29:22.761316 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:29:22.761318 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:29:22.762514 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:29:22.762518 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:29:22.763171 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:29:22.763172 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:29:22.763293 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:29:22.764026 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:29:22.764032 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:29:22.765948 (+ 1916us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:29:22.768318 (+ 2370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:29:22.768327 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:29:22.768329 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:29:22.768658 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:29:22.768992 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:29:22.768993 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:29:22.768993 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:29:22.769075 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:29:22.769404 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:29:22.769411 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:29:22.769506 (+ 95us) client_negotiation.cc:769] Sending connection context
0620 09:29:22.769611 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 09:29:22.769777 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1009,"thread_start_us":157,"threads_started":1}
I20260620 09:29:22.770908 32599 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d97072dd4574ae989acbd1f63be2d07 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:29:02 UTC) from {username='kudu', principal='kudu/127.16.43.190@KRBTEST.COM'} at 127.0.0.1:51494
I20260620 09:29:22.771692 32674 tablet_replica.cc:330] stopping tablet replica
I20260620 09:29:22.771812 32674 raft_consensus.cc:2229] T 5d97072dd4574ae989acbd1f63be2d07 P f4e2727ee8d7413a9aa64423a656905a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:29:22.771960 32674 raft_consensus.cc:2258] T 5d97072dd4574ae989acbd1f63be2d07 P f4e2727ee8d7413a9aa64423a656905a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:29:22.772399 32674 ts_tablet_manager.cc:1905] T 5d97072dd4574ae989acbd1f63be2d07 P f4e2727ee8d7413a9aa64423a656905a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:29:22.774760 32674 ts_tablet_manager.cc:1918] T 5d97072dd4574ae989acbd1f63be2d07 P f4e2727ee8d7413a9aa64423a656905a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:29:22.774837 32674 log.cc:1196] T 5d97072dd4574ae989acbd1f63be2d07 P f4e2727ee8d7413a9aa64423a656905a: Deleting WAL directory at /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-0/wal/wals/5d97072dd4574ae989acbd1f63be2d07
I20260620 09:29:22.775102 32674 ts_tablet_manager.cc:1939] T 5d97072dd4574ae989acbd1f63be2d07 P f4e2727ee8d7413a9aa64423a656905a: Deleting consensus metadata
I20260620 09:29:22.775805 32462 catalog_manager.cc:4841] TS f4e2727ee8d7413a9aa64423a656905a (127.16.43.129:42439): tablet 5d97072dd4574ae989acbd1f63be2d07 (table test-table [id=aad942f7ef44461ab8bdf925f59a3116]) successfully deleted
W20260620 09:29:22.834053 32669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:29:22.834338 32669 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:29:22.834384 32669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:29:22.835807 32669 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:29:22.835883 32669 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:29:22.835934 32669 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.43.130
I20260620 09:29:22.837508 32669 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:41053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.130.keytab
--principal=kudu/127.16.43.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.43.130:43087
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.43.130
--webserver_port=40973
--tserver_master_addrs=127.16.43.190:44381
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32669
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.43.130
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:29:22.837808 32669 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:29:22.838081 32669 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:29:22.844126 32679 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:29:22.844138 32682 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:29:22.844138 32680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:29:22.844276 32669 server_base.cc:1034] running on GCE node
I20260620 09:29:22.844726 32669 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:29:22.845082 32669 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:29:22.846279 32669 hybrid_clock.cc:648] HybridClock initialized: now 1781947762846262 us; error 41 us; skew 500 ppm
Jun 20 09:29:22 dist-test-slave-706q krb5kdc[31897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947762, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:29:22.849568 32669 init.cc:377] Logged in from keytab as kudu/127.16.43.130@KRBTEST.COM (short username kudu)
I20260620 09:29:22.850162 32669 webserver.cc:460] Webserver started at http://127.16.43.130:40973/ using document root <none> and password file <none>
I20260620 09:29:22.850432 32669 fs_manager.cc:359] Metadata directory not provided
I20260620 09:29:22.850544 32669 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:29:22.852378 32669 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 09:29:22.853238 32689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:29:22.853575 32669 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:29:22.853729 32669 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-1/wal
uuid: "c87b67aa6f1a434c85643b5029ea8a91"
format_stamp: "Formatted at 2026-06-20 09:29:01 on dist-test-slave-706q"
server_key: "516ea6a6d812ad1e363f11cdfb7b7c73"
server_key_iv: "ca67562decf88946b3c07737280ab04c"
server_key_version: "encryptionkey@0"
I20260620 09:29:22.854212 32669 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:29:22.883087 32669 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:29:22.886277 32669 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:29:22.886451 32669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:29:22.886763 32669 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:29:22.887349 32696 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 09:29:22.889071 32669 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 09:29:22.889163 32669 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260620 09:29:22.889218 32669 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 09:29:22.890081 32669 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 09:29:22.890146 32669 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260620 09:29:22.890312 32696 tablet_bootstrap.cc:492] T 5d97072dd4574ae989acbd1f63be2d07 P c87b67aa6f1a434c85643b5029ea8a91: Bootstrap starting.
I20260620 09:29:22.900710 32669 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.130:43087
I20260620 09:29:22.900863 335 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.130:43087 every 8 connection(s)
I20260620 09:29:22.901458 32669 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-1/data/info.pb
I20260620 09:29:22.904280 32696 log.cc:824] T 5d97072dd4574ae989acbd1f63be2d07 P c87b67aa6f1a434c85643b5029ea8a91: Log is configured to *not* fsync() on all Append() calls
I20260620 09:29:22.905709 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 32669
I20260620 09:29:22.906037 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.43.131:41523
--local_ip_for_outbound_sockets=127.16.43.131
--tserver_master_addrs=127.16.43.190:44381
--webserver_port=34609
--webserver_interface=127.16.43.131
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.131.keytab
--principal=kudu/127.16.43.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.43.148:41053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 09:29:22.907557 32696 tablet_bootstrap.cc:492] T 5d97072dd4574ae989acbd1f63be2d07 P c87b67aa6f1a434c85643b5029ea8a91: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 09:29:22.907990 32696 tablet_bootstrap.cc:492] T 5d97072dd4574ae989acbd1f63be2d07 P c87b67aa6f1a434c85643b5029ea8a91: Bootstrap complete.
Jun 20 09:29:22 dist-test-slave-706q krb5kdc[31897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947762, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.130@KRBTEST.COM for kudu/127.16.43.190@KRBTEST.COM
I20260620 09:29:22.909017 32696 ts_tablet_manager.cc:1397] T 5d97072dd4574ae989acbd1f63be2d07 P c87b67aa6f1a434c85643b5029ea8a91: Time spent bootstrapping tablet: real 0.019s user 0.011s sys 0.007s
I20260620 09:29:22.910869 32696 raft_consensus.cc:348] T 5d97072dd4574ae989acbd1f63be2d07 P c87b67aa6f1a434c85643b5029ea8a91 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "804331f690c6404db86d9c664915e9d2" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 41523 } } peers { permanent_uuid: "f4e2727ee8d7413a9aa64423a656905a" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 42439 } } peers { permanent_uuid: "c87b67aa6f1a434c85643b5029ea8a91" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 43087 } }
I20260620 09:29:22.911316 32696 raft_consensus.cc:729] T 5d97072dd4574ae989acbd1f63be2d07 P c87b67aa6f1a434c85643b5029ea8a91 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c87b67aa6f1a434c85643b5029ea8a91, State: Initialized, Role: FOLLOWER
I20260620 09:29:22.911553 32696 consensus_queue.cc:260] T 5d97072dd4574ae989acbd1f63be2d07 P c87b67aa6f1a434c85643b5029ea8a91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "804331f690c6404db86d9c664915e9d2" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 41523 } } peers { permanent_uuid: "f4e2727ee8d7413a9aa64423a656905a" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 42439 } } peers { permanent_uuid: "c87b67aa6f1a434c85643b5029ea8a91" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 43087 } }
I20260620 09:29:22.911959 32696 ts_tablet_manager.cc:1428] T 5d97072dd4574ae989acbd1f63be2d07 P c87b67aa6f1a434c85643b5029ea8a91: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260620 09:29:22.912963 32668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:29:22.902537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.130:48421 (local address 127.16.43.190:44381)
0620 09:29:22.902612 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0620 09:29:22.902614 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:29:22.902916 (+ 302us) server_negotiation.cc:407] Connection header received
0620 09:29:22.903517 (+ 601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:29:22.903520 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:29:22.903577 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:29:22.903672 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:29:22.904516 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:29:22.905044 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:29:22.906366 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:29:22.906599 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:29:22.906690 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:29:22.909661 (+ 2971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:29:22.909662 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:29:22.909663 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:29:22.909670 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:29:22.911195 (+ 1525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:29:22.911867 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:29:22.911868 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:29:22.911869 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:29:22.911935 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:29:22.912257 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:29:22.912258 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:29:22.912258 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:29:22.912394 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:29:22.912484 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 09:29:22.912765 (+ 281us) server_negotiation.cc:299] Negotiation successful
0620 09:29:22.912861 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 09:29:22.913205 338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:29:22.902507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:44381 (local address 127.16.43.130:48421)
0620 09:29:22.902740 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0620 09:29:22.902781 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 09:29:22.903378 (+ 597us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:29:22.903740 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:29:22.903743 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:29:22.904110 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:29:22.904367 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:29:22.904381 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:29:22.905175 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:29:22.905176 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:29:22.906276 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:29:22.906280 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:29:22.906725 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:29:22.906727 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:29:22.906860 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:29:22.907451 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:29:22.907460 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:29:22.909566 (+ 2106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:29:22.911308 (+ 1742us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:29:22.911316 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:29:22.911319 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:29:22.911770 (+ 451us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:29:22.912029 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:29:22.912029 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:29:22.912030 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:29:22.912184 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:29:22.912467 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:29:22.912473 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:29:22.912631 (+ 158us) client_negotiation.cc:769] Sending connection context
0620 09:29:22.912746 (+ 115us) client_negotiation.cc:240] Negotiation successful
0620 09:29:22.912944 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":134,"threads_started":1}
I20260620 09:29:22.913678 336 heartbeater.cc:347] Connected to a master server at 127.16.43.190:44381
I20260620 09:29:22.913782 336 heartbeater.cc:464] Registering TS with master...
I20260620 09:29:22.913991 336 heartbeater.cc:511] Master 127.16.43.190:44381 requested a full tablet report, sending...
I20260620 09:29:22.914597 32477 ts_manager.cc:194] Registered new tserver with Master: c87b67aa6f1a434c85643b5029ea8a91 (127.16.43.130:43087)
I20260620 09:29:22.915143 32477 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.43.130@KRBTEST.COM'} at 127.16.43.130:48421
I20260620 09:29:22.916074 336 heartbeater.cc:503] Master 127.16.43.190:44381 was elected leader, sending a full tablet report...
I20260620 09:29:22.920585 32672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:29:22.915064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.130:43087 (local address 127.0.0.1:32910)
0620 09:29:22.915488 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0620 09:29:22.915502 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 09:29:22.915595 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:29:22.916106 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:29:22.916107 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:29:22.916110 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:29:22.916162 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:29:22.916164 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:29:22.917688 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:29:22.917690 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:29:22.918948 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:29:22.918951 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:29:22.920305 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:29:22.920306 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:29:22.920377 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:29:22.920399 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 09:29:22.920475 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 09:29:22.920523 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":402}
I20260620 09:29:22.920639 342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:29:22.915309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32910 (local address 127.16.43.130:43087)
0620 09:29:22.915593 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0620 09:29:22.915601 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 09:29:22.915624 (+ 23us) server_negotiation.cc:407] Connection header received
0620 09:29:22.915694 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:29:22.915697 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:29:22.915939 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:29:22.916072 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:29:22.916244 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:29:22.917503 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:29:22.919061 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:29:22.920217 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:29:22.920305 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:29:22.920392 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 09:29:22.920496 (+ 104us) server_negotiation.cc:299] Negotiation successful
0620 09:29:22.920541 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":121,"threads_started":1}
I20260620 09:29:22.921202 32738 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d97072dd4574ae989acbd1f63be2d07 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:29:02 UTC) from {username='kudu', principal='kudu/127.16.43.190@KRBTEST.COM'} at 127.0.0.1:32910
I20260620 09:29:22.921902 343 tablet_replica.cc:330] stopping tablet replica
I20260620 09:29:22.922037 343 raft_consensus.cc:2229] T 5d97072dd4574ae989acbd1f63be2d07 P c87b67aa6f1a434c85643b5029ea8a91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:29:22.922125 343 raft_consensus.cc:2258] T 5d97072dd4574ae989acbd1f63be2d07 P c87b67aa6f1a434c85643b5029ea8a91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:29:22.922564 343 ts_tablet_manager.cc:1905] T 5d97072dd4574ae989acbd1f63be2d07 P c87b67aa6f1a434c85643b5029ea8a91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:29:22.925616 343 ts_tablet_manager.cc:1918] T 5d97072dd4574ae989acbd1f63be2d07 P c87b67aa6f1a434c85643b5029ea8a91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:29:22.925735 343 log.cc:1196] T 5d97072dd4574ae989acbd1f63be2d07 P c87b67aa6f1a434c85643b5029ea8a91: Deleting WAL directory at /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-1/wal/wals/5d97072dd4574ae989acbd1f63be2d07
I20260620 09:29:22.925998 343 ts_tablet_manager.cc:1939] T 5d97072dd4574ae989acbd1f63be2d07 P c87b67aa6f1a434c85643b5029ea8a91: Deleting consensus metadata
I20260620 09:29:22.926781 32464 catalog_manager.cc:4841] TS c87b67aa6f1a434c85643b5029ea8a91 (127.16.43.130:43087): tablet 5d97072dd4574ae989acbd1f63be2d07 (table test-table [id=aad942f7ef44461ab8bdf925f59a3116]) successfully deleted
W20260620 09:29:23.003696 340 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:29:23.003964 340 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:29:23.003991 340 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:29:23.005342 340 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:29:23.005394 340 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:29:23.005429 340 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.43.131
I20260620 09:29:23.007045 340 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:41053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.131.keytab
--principal=kudu/127.16.43.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.43.131:41523
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.43.131
--webserver_port=34609
--tserver_master_addrs=127.16.43.190:44381
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.340
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.43.131
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:29:23.007346 340 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:29:23.007664 340 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:29:23.013721 351 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:29:23.013721 348 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:29:23.013752 349 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:29:23.013988 340 server_base.cc:1034] running on GCE node
I20260620 09:29:23.014297 340 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:29:23.014614 340 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:29:23.015789 340 hybrid_clock.cc:648] HybridClock initialized: now 1781947763015767 us; error 48 us; skew 500 ppm
Jun 20 09:29:23 dist-test-slave-706q krb5kdc[31897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947763, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:29:23.018573 340 init.cc:377] Logged in from keytab as kudu/127.16.43.131@KRBTEST.COM (short username kudu)
I20260620 09:29:23.019136 340 webserver.cc:460] Webserver started at http://127.16.43.131:34609/ using document root <none> and password file <none>
I20260620 09:29:23.019382 340 fs_manager.cc:359] Metadata directory not provided
I20260620 09:29:23.019445 340 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:29:23.021034 340 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:29:23.021744 358 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:29:23.021984 340 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:29:23.022094 340 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-2/wal
uuid: "804331f690c6404db86d9c664915e9d2"
format_stamp: "Formatted at 2026-06-20 09:29:02 on dist-test-slave-706q"
server_key: "76d950b760a461df80d6a522e8c6077e"
server_key_iv: "29895f931e63037c03a0a64c62b1fda6"
server_key_version: "encryptionkey@0"
I20260620 09:29:23.022589 340 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:29:23.051048 340 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:29:23.053251 340 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:29:23.053375 340 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:29:23.053617 340 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:29:23.053972 340 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:29:23.054006 340 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:29:23.054030 340 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:29:23.054123 340 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:29:23.060402 340 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.131:41523
I20260620 09:29:23.060456 471 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.131:41523 every 8 connection(s)
I20260620 09:29:23.060848 340 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-2/data/info.pb
I20260620 09:29:23.063897 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 340
Jun 20 09:29:23 dist-test-slave-706q krb5kdc[31897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947763, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.131@KRBTEST.COM for kudu/127.16.43.190@KRBTEST.COM
I20260620 09:29:23.071346 476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:29:23.064336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47126 (local address 127.16.43.190:44381)
0620 09:29:23.064466 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 09:29:23.064468 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:29:23.064529 (+ 61us) server_negotiation.cc:407] Connection header received
0620 09:29:23.064607 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:29:23.064608 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:29:23.064620 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:29:23.064736 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:29:23.065376 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:29:23.065865 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:29:23.066789 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:29:23.066986 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:29:23.067071 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:29:23.068539 (+ 1468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:29:23.068540 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:29:23.068540 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:29:23.068545 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:29:23.070105 (+ 1560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:29:23.070520 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:29:23.070520 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:29:23.070521 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:29:23.070564 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:29:23.070761 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:29:23.070761 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:29:23.070761 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:29:23.070861 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:29:23.071039 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0620 09:29:23.071094 (+ 55us) server_negotiation.cc:299] Negotiation successful
0620 09:29:23.071200 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":80,"threads_started":1}
I20260620 09:29:23.071472 32668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:29:23.061685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.131:40507 (local address 127.16.43.190:44381)
0620 09:29:23.061754 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0620 09:29:23.061756 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:29:23.062006 (+ 250us) server_negotiation.cc:407] Connection header received
0620 09:29:23.062617 (+ 611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:29:23.062618 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:29:23.062630 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:29:23.062687 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:29:23.063718 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:29:23.064382 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:29:23.065412 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:29:23.065614 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:29:23.065701 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:29:23.068208 (+ 2507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:29:23.068209 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:29:23.068210 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:29:23.068216 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:29:23.069880 (+ 1664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:29:23.070322 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:29:23.070323 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:29:23.070323 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:29:23.070393 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:29:23.070689 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:29:23.070689 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:29:23.070689 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:29:23.070858 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:29:23.070935 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 09:29:23.071280 (+ 345us) server_negotiation.cc:299] Negotiation successful
0620 09:29:23.071347 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 09:29:23.071952 474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:29:23.061707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:44381 (local address 127.16.43.131:40507)
0620 09:29:23.061882 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0620 09:29:23.061918 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 09:29:23.062503 (+ 585us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:29:23.062769 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:29:23.062773 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:29:23.063338 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:29:23.063627 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:29:23.063632 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:29:23.064489 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:29:23.064490 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:29:23.065310 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:29:23.065318 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:29:23.065711 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:29:23.065712 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:29:23.065841 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:29:23.066469 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:29:23.066478 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:29:23.068100 (+ 1622us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:29:23.069960 (+ 1860us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:29:23.069966 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:29:23.069968 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:29:23.070246 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:29:23.070461 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:29:23.070462 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:29:23.070463 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:29:23.070613 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:29:23.070943 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:29:23.070950 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:29:23.071127 (+ 177us) client_negotiation.cc:769] Sending connection context
0620 09:29:23.071247 (+ 120us) client_negotiation.cc:240] Negotiation successful
0620 09:29:23.071708 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":87,"threads_started":1}
I20260620 09:29:23.072360 472 heartbeater.cc:347] Connected to a master server at 127.16.43.190:44381
I20260620 09:29:23.072484 472 heartbeater.cc:464] Registering TS with master...
I20260620 09:29:23.072678 472 heartbeater.cc:511] Master 127.16.43.190:44381 requested a full tablet report, sending...
I20260620 09:29:23.073076 32477 ts_manager.cc:194] Registered new tserver with Master: 804331f690c6404db86d9c664915e9d2 (127.16.43.131:41523)
I20260620 09:29:23.073443 32477 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.43.131@KRBTEST.COM'} at 127.16.43.131:40507
I20260620 09:29:23.083247 16558 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 09:29:23.086444 32668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:29:23.083706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47132 (local address 127.16.43.190:44381)
0620 09:29:23.083772 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0620 09:29:23.083774 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:29:23.083908 (+ 134us) server_negotiation.cc:407] Connection header received
0620 09:29:23.083962 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:29:23.083963 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:29:23.084002 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:29:23.084074 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:29:23.084222 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:29:23.084704 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:29:23.085567 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:29:23.085751 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:29:23.085805 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:29:23.085945 (+ 140us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:29:23.086048 (+ 103us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0620 09:29:23.086358 (+ 310us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:47132: token expired
Metrics: {"server-negotiator.queue_time_us":41}
W20260620 09:29:23.086519 32668 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:47132: token expired
W20260620 09:29:23.086476 32406 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.16.43.190:44381)
I20260620 09:29:23.092578 32668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:29:23.086773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47142 (local address 127.16.43.190:44381)
0620 09:29:23.086808 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0620 09:29:23.086810 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:29:23.086859 (+ 49us) server_negotiation.cc:407] Connection header received
0620 09:29:23.086908 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:29:23.086908 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:29:23.086916 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:29:23.086965 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:29:23.087502 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:29:23.088014 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:29:23.088702 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:29:23.088870 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:29:23.088919 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:29:23.090087 (+ 1168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:29:23.090089 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:29:23.090089 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:29:23.090095 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:29:23.091606 (+ 1511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:29:23.091922 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:29:23.091923 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:29:23.091923 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:29:23.091974 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:29:23.092148 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:29:23.092148 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:29:23.092148 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:29:23.092279 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:29:23.092354 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 09:29:23.092441 (+ 87us) server_negotiation.cc:299] Negotiation successful
0620 09:29:23.092514 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260620 09:29:23.093776 32477 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47142:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 09:29:23.095613 32477 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 09:29:23.098027 32599 tablet_service.cc:1467] Processing CreateTablet for tablet b242f47f6b714327bd03c0dc3b32c374 (DEFAULT_TABLE table=test-table [id=678a4f52a1b64384bbb242f7c574fa21]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:29:23.098063 32738 tablet_service.cc:1467] Processing CreateTablet for tablet b242f47f6b714327bd03c0dc3b32c374 (DEFAULT_TABLE table=test-table [id=678a4f52a1b64384bbb242f7c574fa21]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:29:23.098253 32738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b242f47f6b714327bd03c0dc3b32c374. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:29:23.098255 32599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b242f47f6b714327bd03c0dc3b32c374. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:29:23.099805 32696 tablet_bootstrap.cc:492] T b242f47f6b714327bd03c0dc3b32c374 P c87b67aa6f1a434c85643b5029ea8a91: Bootstrap starting.
I20260620 09:29:23.099956 32557 tablet_bootstrap.cc:492] T b242f47f6b714327bd03c0dc3b32c374 P f4e2727ee8d7413a9aa64423a656905a: Bootstrap starting.
I20260620 09:29:23.100158 32696 tablet_bootstrap.cc:654] T b242f47f6b714327bd03c0dc3b32c374 P c87b67aa6f1a434c85643b5029ea8a91: Neither blocks nor log segments found. Creating new log.
I20260620 09:29:23.100232 32557 tablet_bootstrap.cc:654] T b242f47f6b714327bd03c0dc3b32c374 P f4e2727ee8d7413a9aa64423a656905a: Neither blocks nor log segments found. Creating new log.
I20260620 09:29:23.100780 32696 tablet_bootstrap.cc:492] T b242f47f6b714327bd03c0dc3b32c374 P c87b67aa6f1a434c85643b5029ea8a91: No bootstrap required, opened a new log
I20260620 09:29:23.100834 32696 ts_tablet_manager.cc:1397] T b242f47f6b714327bd03c0dc3b32c374 P c87b67aa6f1a434c85643b5029ea8a91: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260620 09:29:23.100966 32696 raft_consensus.cc:348] T b242f47f6b714327bd03c0dc3b32c374 P c87b67aa6f1a434c85643b5029ea8a91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "804331f690c6404db86d9c664915e9d2" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 41523 } } peers { permanent_uuid: "f4e2727ee8d7413a9aa64423a656905a" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 42439 } } peers { permanent_uuid: "c87b67aa6f1a434c85643b5029ea8a91" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 43087 } }
I20260620 09:29:23.101053 32696 raft_consensus.cc:374] T b242f47f6b714327bd03c0dc3b32c374 P c87b67aa6f1a434c85643b5029ea8a91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:29:23.101111 32696 raft_consensus.cc:729] T b242f47f6b714327bd03c0dc3b32c374 P c87b67aa6f1a434c85643b5029ea8a91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c87b67aa6f1a434c85643b5029ea8a91, State: Initialized, Role: FOLLOWER
I20260620 09:29:23.101182 32696 consensus_queue.cc:260] T b242f47f6b714327bd03c0dc3b32c374 P c87b67aa6f1a434c85643b5029ea8a91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "804331f690c6404db86d9c664915e9d2" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 41523 } } peers { permanent_uuid: "f4e2727ee8d7413a9aa64423a656905a" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 42439 } } peers { permanent_uuid: "c87b67aa6f1a434c85643b5029ea8a91" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 43087 } }
I20260620 09:29:23.101245 32557 tablet_bootstrap.cc:492] T b242f47f6b714327bd03c0dc3b32c374 P f4e2727ee8d7413a9aa64423a656905a: No bootstrap required, opened a new log
I20260620 09:29:23.101284 32557 ts_tablet_manager.cc:1397] T b242f47f6b714327bd03c0dc3b32c374 P f4e2727ee8d7413a9aa64423a656905a: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260620 09:29:23.101297 32696 ts_tablet_manager.cc:1428] T b242f47f6b714327bd03c0dc3b32c374 P c87b67aa6f1a434c85643b5029ea8a91: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 09:29:23.101431 32557 raft_consensus.cc:348] T b242f47f6b714327bd03c0dc3b32c374 P f4e2727ee8d7413a9aa64423a656905a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "804331f690c6404db86d9c664915e9d2" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 41523 } } peers { permanent_uuid: "f4e2727ee8d7413a9aa64423a656905a" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 42439 } } peers { permanent_uuid: "c87b67aa6f1a434c85643b5029ea8a91" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 43087 } }
I20260620 09:29:23.101502 32557 raft_consensus.cc:374] T b242f47f6b714327bd03c0dc3b32c374 P f4e2727ee8d7413a9aa64423a656905a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:29:23.101554 32557 raft_consensus.cc:729] T b242f47f6b714327bd03c0dc3b32c374 P f4e2727ee8d7413a9aa64423a656905a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4e2727ee8d7413a9aa64423a656905a, State: Initialized, Role: FOLLOWER
I20260620 09:29:23.101601 32557 consensus_queue.cc:260] T b242f47f6b714327bd03c0dc3b32c374 P f4e2727ee8d7413a9aa64423a656905a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "804331f690c6404db86d9c664915e9d2" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 41523 } } peers { permanent_uuid: "f4e2727ee8d7413a9aa64423a656905a" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 42439 } } peers { permanent_uuid: "c87b67aa6f1a434c85643b5029ea8a91" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 43087 } }
I20260620 09:29:23.101701 32557 ts_tablet_manager.cc:1428] T b242f47f6b714327bd03c0dc3b32c374 P f4e2727ee8d7413a9aa64423a656905a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 09:29:23.102452 32672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:29:23.097679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.131:41523 (local address 127.0.0.1:56468)
0620 09:29:23.097715 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0620 09:29:23.097727 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 09:29:23.097766 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:29:23.098212 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:29:23.098213 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:29:23.098215 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:29:23.098264 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:29:23.098266 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:29:23.099628 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:29:23.099629 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:29:23.100564 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:29:23.100567 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:29:23.102254 (+ 1687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:29:23.102255 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:29:23.102313 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:29:23.102317 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 09:29:23.102363 (+ 46us) client_negotiation.cc:240] Negotiation successful
0620 09:29:23.102401 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17}
I20260620 09:29:23.102468 480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:29:23.097816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56468 (local address 127.16.43.131:41523)
0620 09:29:23.098013 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 09:29:23.098018 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:29:23.098029 (+ 11us) server_negotiation.cc:407] Connection header received
0620 09:29:23.098071 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:29:23.098073 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:29:23.098090 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:29:23.098341 (+ 251us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:29:23.098424 (+ 83us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:29:23.099496 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:29:23.101377 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:29:23.102177 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:29:23.102249 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:29:23.102313 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 09:29:23.102377 (+ 64us) server_negotiation.cc:299] Negotiation successful
0620 09:29:23.102407 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":93,"threads_started":1}
I20260620 09:29:23.103271 406 tablet_service.cc:1467] Processing CreateTablet for tablet b242f47f6b714327bd03c0dc3b32c374 (DEFAULT_TABLE table=test-table [id=678a4f52a1b64384bbb242f7c574fa21]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:29:23.103641 406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b242f47f6b714327bd03c0dc3b32c374. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:29:23.106546 483 tablet_bootstrap.cc:492] T b242f47f6b714327bd03c0dc3b32c374 P 804331f690c6404db86d9c664915e9d2: Bootstrap starting.
I20260620 09:29:23.107161 483 tablet_bootstrap.cc:654] T b242f47f6b714327bd03c0dc3b32c374 P 804331f690c6404db86d9c664915e9d2: Neither blocks nor log segments found. Creating new log.
I20260620 09:29:23.107769 483 log.cc:824] T b242f47f6b714327bd03c0dc3b32c374 P 804331f690c6404db86d9c664915e9d2: Log is configured to *not* fsync() on all Append() calls
I20260620 09:29:23.108628 483 tablet_bootstrap.cc:492] T b242f47f6b714327bd03c0dc3b32c374 P 804331f690c6404db86d9c664915e9d2: No bootstrap required, opened a new log
I20260620 09:29:23.108693 483 ts_tablet_manager.cc:1397] T b242f47f6b714327bd03c0dc3b32c374 P 804331f690c6404db86d9c664915e9d2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 09:29:23.110091 483 raft_consensus.cc:348] T b242f47f6b714327bd03c0dc3b32c374 P 804331f690c6404db86d9c664915e9d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "804331f690c6404db86d9c664915e9d2" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 41523 } } peers { permanent_uuid: "f4e2727ee8d7413a9aa64423a656905a" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 42439 } } peers { permanent_uuid: "c87b67aa6f1a434c85643b5029ea8a91" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 43087 } }
I20260620 09:29:23.110239 483 raft_consensus.cc:374] T b242f47f6b714327bd03c0dc3b32c374 P 804331f690c6404db86d9c664915e9d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:29:23.110306 483 raft_consensus.cc:729] T b242f47f6b714327bd03c0dc3b32c374 P 804331f690c6404db86d9c664915e9d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 804331f690c6404db86d9c664915e9d2, State: Initialized, Role: FOLLOWER
I20260620 09:29:23.110452 483 consensus_queue.cc:260] T b242f47f6b714327bd03c0dc3b32c374 P 804331f690c6404db86d9c664915e9d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "804331f690c6404db86d9c664915e9d2" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 41523 } } peers { permanent_uuid: "f4e2727ee8d7413a9aa64423a656905a" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 42439 } } peers { permanent_uuid: "c87b67aa6f1a434c85643b5029ea8a91" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 43087 } }
I20260620 09:29:23.110740 483 ts_tablet_manager.cc:1428] T b242f47f6b714327bd03c0dc3b32c374 P 804331f690c6404db86d9c664915e9d2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 09:29:23.110805 472 heartbeater.cc:503] Master 127.16.43.190:44381 was elected leader, sending a full tablet report...
W20260620 09:29:23.152344 337 tablet.cc:2369] T b242f47f6b714327bd03c0dc3b32c374 P c87b67aa6f1a434c85643b5029ea8a91: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 09:29:23.240234 32666 tablet.cc:2369] T b242f47f6b714327bd03c0dc3b32c374 P f4e2727ee8d7413a9aa64423a656905a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 09:29:23.261909 341 raft_consensus.cc:482] T b242f47f6b714327bd03c0dc3b32c374 P c87b67aa6f1a434c85643b5029ea8a91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:29:23.262041 341 raft_consensus.cc:504] T b242f47f6b714327bd03c0dc3b32c374 P c87b67aa6f1a434c85643b5029ea8a91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "804331f690c6404db86d9c664915e9d2" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 41523 } } peers { permanent_uuid: "f4e2727ee8d7413a9aa64423a656905a" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 42439 } } peers { permanent_uuid: "c87b67aa6f1a434c85643b5029ea8a91" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 43087 } }
I20260620 09:29:23.262404 341 leader_election.cc:288] T b242f47f6b714327bd03c0dc3b32c374 P c87b67aa6f1a434c85643b5029ea8a91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 804331f690c6404db86d9c664915e9d2 (127.16.43.131:41523), f4e2727ee8d7413a9aa64423a656905a (127.16.43.129:42439)
I20260620 09:29:23.266430 338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:29:23.262694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.131:41523 (local address 127.16.43.130:50193)
0620 09:29:23.262757 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0620 09:29:23.262771 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 09:29:23.262871 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:29:23.263060 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:29:23.263061 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:29:23.263063 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:29:23.263145 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:29:23.263152 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:29:23.264111 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:29:23.264112 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:29:23.265376 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:29:23.265379 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:29:23.266168 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:29:23.266169 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:29:23.266240 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:29:23.266244 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 09:29:23.266289 (+ 45us) client_negotiation.cc:240] Negotiation successful
0620 09:29:23.266348 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":2}
I20260620 09:29:23.266647 486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:29:23.262824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.129:42439 (local address 127.16.43.130:59813)
0620 09:29:23.262988 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 09:29:23.262998 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 09:29:23.263063 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:29:23.263295 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:29:23.263295 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:29:23.263297 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:29:23.263357 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:29:23.263358 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:29:23.264245 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:29:23.264246 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:29:23.265376 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:29:23.265379 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:29:23.266372 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:29:23.266373 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:29:23.266433 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:29:23.266437 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 09:29:23.266493 (+ 56us) client_negotiation.cc:240] Negotiation successful
0620 09:29:23.266534 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"mutex_wait_us":6,"thread_start_us":115,"threads_started":1}
I20260620 09:29:23.266641 32673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:29:23.262938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.130:59813 (local address 127.16.43.129:42439)
0620 09:29:23.262988 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 09:29:23.262990 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:29:23.263042 (+ 52us) server_negotiation.cc:407] Connection header received
0620 09:29:23.263161 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:29:23.263163 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:29:23.263208 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:29:23.263287 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:29:23.263436 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:29:23.264178 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:29:23.265501 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:29:23.266257 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:29:23.266353 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:29:23.266427 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 09:29:23.266518 (+ 91us) server_negotiation.cc:299] Negotiation successful
0620 09:29:23.266574 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 09:29:23.266446 480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:29:23.262857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.130:50193 (local address 127.16.43.131:41523)
0620 09:29:23.262916 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0620 09:29:23.262919 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:29:23.262930 (+ 11us) server_negotiation.cc:407] Connection header received
0620 09:29:23.262966 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:29:23.262967 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:29:23.262979 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:29:23.263054 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:29:23.263232 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:29:23.264031 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:29:23.265496 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:29:23.266076 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:29:23.266168 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:29:23.266180 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0620 09:29:23.266326 (+ 146us) server_negotiation.cc:299] Negotiation successful
0620 09:29:23.266370 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 09:29:23.267072 426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b242f47f6b714327bd03c0dc3b32c374" candidate_uuid: "c87b67aa6f1a434c85643b5029ea8a91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "804331f690c6404db86d9c664915e9d2" is_pre_election: true
I20260620 09:29:23.267129 32619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b242f47f6b714327bd03c0dc3b32c374" candidate_uuid: "c87b67aa6f1a434c85643b5029ea8a91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4e2727ee8d7413a9aa64423a656905a" is_pre_election: true
I20260620 09:29:23.267244 426 raft_consensus.cc:2454] T b242f47f6b714327bd03c0dc3b32c374 P 804331f690c6404db86d9c664915e9d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c87b67aa6f1a434c85643b5029ea8a91 in term 0.
I20260620 09:29:23.267306 32619 raft_consensus.cc:2454] T b242f47f6b714327bd03c0dc3b32c374 P f4e2727ee8d7413a9aa64423a656905a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c87b67aa6f1a434c85643b5029ea8a91 in term 0.
I20260620 09:29:23.267510 32691 leader_election.cc:302] T b242f47f6b714327bd03c0dc3b32c374 P c87b67aa6f1a434c85643b5029ea8a91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 804331f690c6404db86d9c664915e9d2, c87b67aa6f1a434c85643b5029ea8a91; no voters:
I20260620 09:29:23.267690 341 raft_consensus.cc:2781] T b242f47f6b714327bd03c0dc3b32c374 P c87b67aa6f1a434c85643b5029ea8a91 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 09:29:23.267763 341 raft_consensus.cc:482] T b242f47f6b714327bd03c0dc3b32c374 P c87b67aa6f1a434c85643b5029ea8a91 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 09:29:23.267794 341 raft_consensus.cc:3037] T b242f47f6b714327bd03c0dc3b32c374 P c87b67aa6f1a434c85643b5029ea8a91 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:29:23.268510 341 raft_consensus.cc:504] T b242f47f6b714327bd03c0dc3b32c374 P c87b67aa6f1a434c85643b5029ea8a91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "804331f690c6404db86d9c664915e9d2" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 41523 } } peers { permanent_uuid: "f4e2727ee8d7413a9aa64423a656905a" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 42439 } } peers { permanent_uuid: "c87b67aa6f1a434c85643b5029ea8a91" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 43087 } }
I20260620 09:29:23.268646 341 leader_election.cc:288] T b242f47f6b714327bd03c0dc3b32c374 P c87b67aa6f1a434c85643b5029ea8a91 [CANDIDATE]: Term 1 election: Requested vote from peers 804331f690c6404db86d9c664915e9d2 (127.16.43.131:41523), f4e2727ee8d7413a9aa64423a656905a (127.16.43.129:42439)
I20260620 09:29:23.268810 426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b242f47f6b714327bd03c0dc3b32c374" candidate_uuid: "c87b67aa6f1a434c85643b5029ea8a91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "804331f690c6404db86d9c664915e9d2"
I20260620 09:29:23.268843 32619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b242f47f6b714327bd03c0dc3b32c374" candidate_uuid: "c87b67aa6f1a434c85643b5029ea8a91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4e2727ee8d7413a9aa64423a656905a"
I20260620 09:29:23.268913 32619 raft_consensus.cc:3037] T b242f47f6b714327bd03c0dc3b32c374 P f4e2727ee8d7413a9aa64423a656905a [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:29:23.268915 426 raft_consensus.cc:3037] T b242f47f6b714327bd03c0dc3b32c374 P 804331f690c6404db86d9c664915e9d2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:29:23.269552 32619 raft_consensus.cc:2454] T b242f47f6b714327bd03c0dc3b32c374 P f4e2727ee8d7413a9aa64423a656905a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c87b67aa6f1a434c85643b5029ea8a91 in term 1.
I20260620 09:29:23.269717 32691 leader_election.cc:302] T b242f47f6b714327bd03c0dc3b32c374 P c87b67aa6f1a434c85643b5029ea8a91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c87b67aa6f1a434c85643b5029ea8a91, f4e2727ee8d7413a9aa64423a656905a; no voters:
I20260620 09:29:23.269738 426 raft_consensus.cc:2454] T b242f47f6b714327bd03c0dc3b32c374 P 804331f690c6404db86d9c664915e9d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c87b67aa6f1a434c85643b5029ea8a91 in term 1.
I20260620 09:29:23.269804 341 raft_consensus.cc:2781] T b242f47f6b714327bd03c0dc3b32c374 P c87b67aa6f1a434c85643b5029ea8a91 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:29:23.269960 341 raft_consensus.cc:686] T b242f47f6b714327bd03c0dc3b32c374 P c87b67aa6f1a434c85643b5029ea8a91 [term 1 LEADER]: Becoming Leader. State: Replica: c87b67aa6f1a434c85643b5029ea8a91, State: Running, Role: LEADER
I20260620 09:29:23.270035 341 consensus_queue.cc:237] T b242f47f6b714327bd03c0dc3b32c374 P c87b67aa6f1a434c85643b5029ea8a91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "804331f690c6404db86d9c664915e9d2" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 41523 } } peers { permanent_uuid: "f4e2727ee8d7413a9aa64423a656905a" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 42439 } } peers { permanent_uuid: "c87b67aa6f1a434c85643b5029ea8a91" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 43087 } }
I20260620 09:29:23.270869 32476 catalog_manager.cc:5495] T b242f47f6b714327bd03c0dc3b32c374 P c87b67aa6f1a434c85643b5029ea8a91 reported cstate change: term changed from 0 to 1, leader changed from <none> to c87b67aa6f1a434c85643b5029ea8a91 (127.16.43.130). New cstate: current_term: 1 leader_uuid: "c87b67aa6f1a434c85643b5029ea8a91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "804331f690c6404db86d9c664915e9d2" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 41523 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f4e2727ee8d7413a9aa64423a656905a" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 42439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c87b67aa6f1a434c85643b5029ea8a91" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 43087 } health_report { overall_health: HEALTHY } } }
I20260620 09:29:23.286484 342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:29:23.283708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32922 (local address 127.16.43.130:43087)
0620 09:29:23.283772 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 09:29:23.283774 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:29:23.283782 (+ 8us) server_negotiation.cc:407] Connection header received
0620 09:29:23.283879 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:29:23.283880 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:29:23.283897 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:29:23.283958 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:29:23.284157 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:29:23.284630 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:29:23.285675 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:29:23.285876 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:29:23.285946 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:29:23.286138 (+ 192us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:29:23.286281 (+ 143us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:29:23.286347 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 09:29:23.286390 (+ 43us) server_negotiation.cc:299] Negotiation successful
0620 09:29:23.286413 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 09:29:23.288270 32619 raft_consensus.cc:1261] T b242f47f6b714327bd03c0dc3b32c374 P f4e2727ee8d7413a9aa64423a656905a [term 1 FOLLOWER]: Refusing update from remote peer c87b67aa6f1a434c85643b5029ea8a91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:29:23.288302 426 raft_consensus.cc:1261] T b242f47f6b714327bd03c0dc3b32c374 P 804331f690c6404db86d9c664915e9d2 [term 1 FOLLOWER]: Refusing update from remote peer c87b67aa6f1a434c85643b5029ea8a91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:29:23.288641 341 consensus_queue.cc:1035] T b242f47f6b714327bd03c0dc3b32c374 P c87b67aa6f1a434c85643b5029ea8a91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "804331f690c6404db86d9c664915e9d2" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 41523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:29:23.288796 487 consensus_queue.cc:1035] T b242f47f6b714327bd03c0dc3b32c374 P c87b67aa6f1a434c85643b5029ea8a91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4e2727ee8d7413a9aa64423a656905a" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 42439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:29:23.294231 32477 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47142:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:29:23.294320 32477 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47142:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:29:23.295109 32477 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c18c7503906044e3b00e5afc76fde6c0: Sending DeleteTablet for 3 replicas of tablet b242f47f6b714327bd03c0dc3b32c374
I20260620 09:29:23.295445 32738 tablet_service.cc:1514] Processing DeleteTablet for tablet b242f47f6b714327bd03c0dc3b32c374 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:29:23 UTC) from {username='kudu', principal='kudu/127.16.43.190@KRBTEST.COM'} at 127.0.0.1:32910
I20260620 09:29:23.295570 343 tablet_replica.cc:330] T b242f47f6b714327bd03c0dc3b32c374 P c87b67aa6f1a434c85643b5029ea8a91: stopping tablet replica
I20260620 09:29:23.295620 343 raft_consensus.cc:2229] T b242f47f6b714327bd03c0dc3b32c374 P c87b67aa6f1a434c85643b5029ea8a91 [term 1 LEADER]: Raft consensus shutting down.
I20260620 09:29:23.295722 343 raft_consensus.cc:2258] T b242f47f6b714327bd03c0dc3b32c374 P c87b67aa6f1a434c85643b5029ea8a91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:29:23.295706 406 tablet_service.cc:1514] Processing DeleteTablet for tablet b242f47f6b714327bd03c0dc3b32c374 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:29:23 UTC) from {username='kudu', principal='kudu/127.16.43.190@KRBTEST.COM'} at 127.0.0.1:56468
I20260620 09:29:23.295922 32599 tablet_service.cc:1514] Processing DeleteTablet for tablet b242f47f6b714327bd03c0dc3b32c374 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:29:23 UTC) from {username='kudu', principal='kudu/127.16.43.190@KRBTEST.COM'} at 127.0.0.1:51494
I20260620 09:29:23.296041 343 ts_tablet_manager.cc:1905] T b242f47f6b714327bd03c0dc3b32c374 P c87b67aa6f1a434c85643b5029ea8a91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:29:23.296164 500 tablet_replica.cc:330] T b242f47f6b714327bd03c0dc3b32c374 P f4e2727ee8d7413a9aa64423a656905a: stopping tablet replica
I20260620 09:29:23.296182 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 32515
I20260620 09:29:23.296283 500 raft_consensus.cc:2229] T b242f47f6b714327bd03c0dc3b32c374 P f4e2727ee8d7413a9aa64423a656905a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:29:23.296375 500 raft_consensus.cc:2258] T b242f47f6b714327bd03c0dc3b32c374 P f4e2727ee8d7413a9aa64423a656905a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:29:23.297766 343 ts_tablet_manager.cc:1918] T b242f47f6b714327bd03c0dc3b32c374 P c87b67aa6f1a434c85643b5029ea8a91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:29:23.297842 343 log.cc:1196] T b242f47f6b714327bd03c0dc3b32c374 P c87b67aa6f1a434c85643b5029ea8a91: Deleting WAL directory at /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-1/wal/wals/b242f47f6b714327bd03c0dc3b32c374
I20260620 09:29:23.298058 343 ts_tablet_manager.cc:1939] T b242f47f6b714327bd03c0dc3b32c374 P c87b67aa6f1a434c85643b5029ea8a91: Deleting consensus metadata
I20260620 09:29:23.298373 32464 catalog_manager.cc:4841] TS c87b67aa6f1a434c85643b5029ea8a91 (127.16.43.130:43087): tablet b242f47f6b714327bd03c0dc3b32c374 (table test-table [id=678a4f52a1b64384bbb242f7c574fa21]) successfully deleted
I20260620 09:29:23.303746 499 tablet_replica.cc:330] T b242f47f6b714327bd03c0dc3b32c374 P 804331f690c6404db86d9c664915e9d2: stopping tablet replica
I20260620 09:29:23.303951 499 raft_consensus.cc:2229] T b242f47f6b714327bd03c0dc3b32c374 P 804331f690c6404db86d9c664915e9d2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:29:23.304116 499 raft_consensus.cc:2258] T b242f47f6b714327bd03c0dc3b32c374 P 804331f690c6404db86d9c664915e9d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:29:23.304735 499 ts_tablet_manager.cc:1905] T b242f47f6b714327bd03c0dc3b32c374 P 804331f690c6404db86d9c664915e9d2: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 09:29:23.304960 32462 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.43.129:42439 (error 108)
W20260620 09:29:23.305749 32672 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:29:23.305371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.129:42439 (local address 127.0.0.1:51500)
0620 09:29:23.305441 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0620 09:29:23.305457 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 09:29:23.305530 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:29:23.305704 (+ 174us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.129:42439: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":45}
W20260620 09:29:23.305881 32462 catalog_manager.cc:4584] TS f4e2727ee8d7413a9aa64423a656905a (127.16.43.129:42439): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b242f47f6b714327bd03c0dc3b32c374: Network error: Client connection negotiation failed: client connection to 127.16.43.129:42439: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 09:29:23.306777 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 32669
I20260620 09:29:23.306782 499 ts_tablet_manager.cc:1918] T b242f47f6b714327bd03c0dc3b32c374 P 804331f690c6404db86d9c664915e9d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:29:23.311342 499 log.cc:1196] T b242f47f6b714327bd03c0dc3b32c374 P 804331f690c6404db86d9c664915e9d2: Deleting WAL directory at /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-2/wal/wals/b242f47f6b714327bd03c0dc3b32c374
I20260620 09:29:23.311566 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 340
I20260620 09:29:23.311769 499 ts_tablet_manager.cc:1939] T b242f47f6b714327bd03c0dc3b32c374 P 804331f690c6404db86d9c664915e9d2: Deleting consensus metadata
I20260620 09:29:23.316054 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 32444
I20260620 09:29:23.316248 32672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:29:23.316079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.131:41523 (local address 127.0.0.1:56482)
0620 09:29:23.316143 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0620 09:29:23.316201 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.131:41523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260620 09:29:43.321818 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.43.190:44381
--webserver_interface=127.16.43.190
--webserver_port=46375
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.190.keytab
--principal=kudu/127.16.43.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.43.148:41053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.43.190:44381
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:29:43.410082 502 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:29:43.410354 502 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:29:43.410391 502 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:29:43.411593 502 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 09:29:43.411661 502 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 09:29:43.411682 502 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 09:29:43.411700 502 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:29:43.411733 502 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:29:43.411757 502 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:29:43.411783 502 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:29:43.411806 502 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:29:43.413290 502 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:41053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.16.43.190:44381
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.190.keytab
--principal=kudu/127.16.43.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.43.190:44381
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.43.190
--webserver_port=46375
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.502
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:29:43.413569 502 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:29:43.413808 502 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:29:43.419415 510 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:29:43.419476 508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:29:43.419611 502 server_base.cc:1034] running on GCE node
W20260620 09:29:43.419461 507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:29:43.419982 502 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:29:43.420245 502 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:29:43.421408 502 hybrid_clock.cc:648] HybridClock initialized: now 1781947783421398 us; error 34 us; skew 500 ppm
Jun 20 09:29:43 dist-test-slave-706q krb5kdc[31897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947783, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:29:43.424226 502 init.cc:377] Logged in from keytab as kudu/127.16.43.190@KRBTEST.COM (short username kudu)
I20260620 09:29:43.424691 502 webserver.cc:460] Webserver started at http://127.16.43.190:46375/ using document root <none> and password file <none>
I20260620 09:29:43.424898 502 fs_manager.cc:359] Metadata directory not provided
I20260620 09:29:43.424940 502 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:29:43.426239 502 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:29:43.427019 517 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:29:43.427215 502 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 09:29:43.427310 502 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/master-0/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/master-0/wal
uuid: "c18c7503906044e3b00e5afc76fde6c0"
format_stamp: "Formatted at 2026-06-20 09:29:01 on dist-test-slave-706q"
server_key: "8460fbc73d47b6d32e6e6108117f9d71"
server_key_iv: "bba2020cb7145bbd5a4d8e9b74f35dab"
server_key_version: "encryptionkey@0"
I20260620 09:29:43.427724 502 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:29:43.436868 502 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:29:43.439065 502 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:29:43.439195 502 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:29:43.443388 502 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.190:44381
I20260620 09:29:43.443460 569 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.190:44381 every 8 connection(s)
I20260620 09:29:43.443887 502 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
I20260620 09:29:43.445519 570 sys_catalog.cc:263] Verifying existing consensus state
I20260620 09:29:43.446251 570 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c18c7503906044e3b00e5afc76fde6c0: Bootstrap starting.
I20260620 09:29:43.447255 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 502
I20260620 09:29:43.447665 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.43.129:42439
--local_ip_for_outbound_sockets=127.16.43.129
--tserver_master_addrs=127.16.43.190:44381
--webserver_port=36881
--webserver_interface=127.16.43.129
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.129.keytab
--principal=kudu/127.16.43.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.43.148:41053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 09:29:43.456199 570 log.cc:824] T 00000000000000000000000000000000 P c18c7503906044e3b00e5afc76fde6c0: Log is configured to *not* fsync() on all Append() calls
I20260620 09:29:43.458586 570 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c18c7503906044e3b00e5afc76fde6c0: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 09:29:43.458917 570 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c18c7503906044e3b00e5afc76fde6c0: Bootstrap complete.
I20260620 09:29:43.461293 570 raft_consensus.cc:348] T 00000000000000000000000000000000 P c18c7503906044e3b00e5afc76fde6c0 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c18c7503906044e3b00e5afc76fde6c0" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 44381 } }
I20260620 09:29:43.461563 570 raft_consensus.cc:729] T 00000000000000000000000000000000 P c18c7503906044e3b00e5afc76fde6c0 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: c18c7503906044e3b00e5afc76fde6c0, State: Initialized, Role: FOLLOWER
I20260620 09:29:43.461778 570 consensus_queue.cc:260] T 00000000000000000000000000000000 P c18c7503906044e3b00e5afc76fde6c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c18c7503906044e3b00e5afc76fde6c0" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 44381 } }
I20260620 09:29:43.461900 570 raft_consensus.cc:388] T 00000000000000000000000000000000 P c18c7503906044e3b00e5afc76fde6c0 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:29:43.461956 570 raft_consensus.cc:482] T 00000000000000000000000000000000 P c18c7503906044e3b00e5afc76fde6c0 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:29:43.462005 570 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c18c7503906044e3b00e5afc76fde6c0 [term 2 FOLLOWER]: Advancing to term 3
I20260620 09:29:43.463068 570 raft_consensus.cc:504] T 00000000000000000000000000000000 P c18c7503906044e3b00e5afc76fde6c0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c18c7503906044e3b00e5afc76fde6c0" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 44381 } }
I20260620 09:29:43.463194 570 leader_election.cc:302] T 00000000000000000000000000000000 P c18c7503906044e3b00e5afc76fde6c0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c18c7503906044e3b00e5afc76fde6c0; no voters:
I20260620 09:29:43.463395 570 leader_election.cc:288] T 00000000000000000000000000000000 P c18c7503906044e3b00e5afc76fde6c0 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 09:29:43.463478 574 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c18c7503906044e3b00e5afc76fde6c0 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 09:29:43.463696 570 sys_catalog.cc:564] T 00000000000000000000000000000000 P c18c7503906044e3b00e5afc76fde6c0 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:29:43.463706 574 raft_consensus.cc:686] T 00000000000000000000000000000000 P c18c7503906044e3b00e5afc76fde6c0 [term 3 LEADER]: Becoming Leader. State: Replica: c18c7503906044e3b00e5afc76fde6c0, State: Running, Role: LEADER
I20260620 09:29:43.463986 574 consensus_queue.cc:237] T 00000000000000000000000000000000 P c18c7503906044e3b00e5afc76fde6c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c18c7503906044e3b00e5afc76fde6c0" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 44381 } }
I20260620 09:29:43.464298 575 sys_catalog.cc:455] T 00000000000000000000000000000000 P c18c7503906044e3b00e5afc76fde6c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "c18c7503906044e3b00e5afc76fde6c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c18c7503906044e3b00e5afc76fde6c0" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 44381 } } }
I20260620 09:29:43.464403 575 sys_catalog.cc:458] T 00000000000000000000000000000000 P c18c7503906044e3b00e5afc76fde6c0 [sys.catalog]: This master's current role is: LEADER
I20260620 09:29:43.464324 576 sys_catalog.cc:455] T 00000000000000000000000000000000 P c18c7503906044e3b00e5afc76fde6c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c18c7503906044e3b00e5afc76fde6c0. Latest consensus state: current_term: 3 leader_uuid: "c18c7503906044e3b00e5afc76fde6c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c18c7503906044e3b00e5afc76fde6c0" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 44381 } } }
I20260620 09:29:43.464506 576 sys_catalog.cc:458] T 00000000000000000000000000000000 P c18c7503906044e3b00e5afc76fde6c0 [sys.catalog]: This master's current role is: LEADER
I20260620 09:29:43.465626 590 catalog_manager.cc:1259] Loaded cluster ID: 6ef750a9a6cc451da9eb3ed2ee1c6e85
I20260620 09:29:43.465677 590 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c18c7503906044e3b00e5afc76fde6c0: loading cluster ID for follower catalog manager: success
I20260620 09:29:43.466910 590 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c18c7503906044e3b00e5afc76fde6c0: acquiring CA information for follower catalog manager: success
F20260620 09:29:43.467334 590 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f32dc8659ad google::LogMessage::Fail() at ??:0
@ 0x7f32dc86888a google::LogMessage::SendToLog() at ??:0
@ 0x7f32dc8654de google::LogMessage::Flush() at ??:0
@ 0x7f32dc866a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f32defd2104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f32e47be6b9 kudu::master::CatalogManager::LoadTspkEntries() at ??:0
@ 0x7f32e47be989 kudu::master::CatalogManager::PrepareFollowerTokenVerifier() at ??:0
@ 0x7f32e47bee4e kudu::master::CatalogManager::PrepareFollower() at ??:0
@ 0x7f32e47c18c2 kudu::master::CatalogManagerBgTasks::Run() at ??:0
@ 0x7f32de728954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f32df58c6db start_thread at ??:0
@ 0x7f32db36361f clone at ??:0
W20260620 09:29:43.551107 572 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:29:43.551311 572 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:29:43.551335 572 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:29:43.552552 572 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:29:43.552596 572 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:29:43.552623 572 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.43.129
I20260620 09:29:43.554163 572 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:41053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.129.keytab
--principal=kudu/127.16.43.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.43.129:42439
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.43.129
--webserver_port=36881
--tserver_master_addrs=127.16.43.190:44381
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.572
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.43.129
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:29:43.554415 572 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:29:43.554661 572 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:29:43.559980 609 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:29:43.560045 608 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:29:43.560165 572 server_base.cc:1034] running on GCE node
W20260620 09:29:43.560235 611 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:29:43.560534 572 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:29:43.560799 572 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:29:43.561995 572 hybrid_clock.cc:648] HybridClock initialized: now 1781947783561984 us; error 34 us; skew 500 ppm
Jun 20 09:29:43 dist-test-slave-706q krb5kdc[31897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947783, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:29:43.564724 572 init.cc:377] Logged in from keytab as kudu/127.16.43.129@KRBTEST.COM (short username kudu)
I20260620 09:29:43.565214 572 webserver.cc:460] Webserver started at http://127.16.43.129:36881/ using document root <none> and password file <none>
I20260620 09:29:43.565441 572 fs_manager.cc:359] Metadata directory not provided
I20260620 09:29:43.565511 572 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:29:43.566931 572 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:29:43.567729 618 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:29:43.567896 572 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 09:29:43.568001 572 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-0/wal
uuid: "f4e2727ee8d7413a9aa64423a656905a"
format_stamp: "Formatted at 2026-06-20 09:29:01 on dist-test-slave-706q"
server_key: "9727756739f560c432eeb7b0db23a7d7"
server_key_iv: "c8712d4e099cb95692802b4ce558e040"
server_key_version: "encryptionkey@0"
I20260620 09:29:43.568394 572 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:29:43.577761 572 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:29:43.580031 572 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:29:43.580195 572 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:29:43.580427 572 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:29:43.580847 625 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 09:29:43.582851 572 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 09:29:43.582944 572 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260620 09:29:43.583002 572 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 09:29:43.583815 572 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 09:29:43.583880 572 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.001s sys 0.000s
I20260620 09:29:43.583933 625 tablet_bootstrap.cc:492] T b242f47f6b714327bd03c0dc3b32c374 P f4e2727ee8d7413a9aa64423a656905a: Bootstrap starting.
I20260620 09:29:43.591838 572 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.129:42439
I20260620 09:29:43.592249 572 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-0/data/info.pb
I20260620 09:29:43.592847 732 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.129:42439 every 8 connection(s)
I20260620 09:29:43.593822 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 572
I20260620 09:29:43.594169 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.43.130:43087
--local_ip_for_outbound_sockets=127.16.43.130
--tserver_master_addrs=127.16.43.190:44381
--webserver_port=40973
--webserver_interface=127.16.43.130
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.130.keytab
--principal=kudu/127.16.43.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.43.148:41053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 09:29:43.595122 625 log.cc:824] T b242f47f6b714327bd03c0dc3b32c374 P f4e2727ee8d7413a9aa64423a656905a: Log is configured to *not* fsync() on all Append() calls
I20260620 09:29:43.600145 625 tablet_bootstrap.cc:492] T b242f47f6b714327bd03c0dc3b32c374 P f4e2727ee8d7413a9aa64423a656905a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 09:29:43.600481 625 tablet_bootstrap.cc:492] T b242f47f6b714327bd03c0dc3b32c374 P f4e2727ee8d7413a9aa64423a656905a: Bootstrap complete.
I20260620 09:29:43.601719 625 ts_tablet_manager.cc:1397] T b242f47f6b714327bd03c0dc3b32c374 P f4e2727ee8d7413a9aa64423a656905a: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.011s
I20260620 09:29:43.603209 625 raft_consensus.cc:348] T b242f47f6b714327bd03c0dc3b32c374 P f4e2727ee8d7413a9aa64423a656905a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "804331f690c6404db86d9c664915e9d2" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 41523 } } peers { permanent_uuid: "f4e2727ee8d7413a9aa64423a656905a" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 42439 } } peers { permanent_uuid: "c87b67aa6f1a434c85643b5029ea8a91" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 43087 } }
I20260620 09:29:43.603582 625 raft_consensus.cc:729] T b242f47f6b714327bd03c0dc3b32c374 P f4e2727ee8d7413a9aa64423a656905a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4e2727ee8d7413a9aa64423a656905a, State: Initialized, Role: FOLLOWER
I20260620 09:29:43.603721 625 consensus_queue.cc:260] T b242f47f6b714327bd03c0dc3b32c374 P f4e2727ee8d7413a9aa64423a656905a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "804331f690c6404db86d9c664915e9d2" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 41523 } } peers { permanent_uuid: "f4e2727ee8d7413a9aa64423a656905a" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 42439 } } peers { permanent_uuid: "c87b67aa6f1a434c85643b5029ea8a91" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 43087 } }
I20260620 09:29:43.604010 625 ts_tablet_manager.cc:1428] T b242f47f6b714327bd03c0dc3b32c374 P f4e2727ee8d7413a9aa64423a656905a: Time spent starting tablet: real 0.002s user 0.001s sys 0.003s
W20260620 09:29:43.665089 735 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:29:43.592970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:44381 (local address 127.16.43.129:50645)
0620 09:29:43.593170 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0620 09:29:43.593214 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 09:29:43.593805 (+ 591us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:29:43.664880 (+ 71075us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:44381: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":123,"threads_started":1}
W20260620 09:29:43.665369 733 heartbeater.cc:644] Failed to heartbeat to 127.16.43.190:44381 (0 consecutive failures): Network error: Failed to ping master at 127.16.43.190:44381: Client connection negotiation failed: client connection to 127.16.43.190:44381: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 09:29:43.665791 735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:29:43.665635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:44381 (local address 127.16.43.129:47185)
0620 09:29:43.665713 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0620 09:29:43.665759 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:44381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260620 09:29:43.666288 735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:29:43.666146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:44381 (local address 127.16.43.129:53057)
0620 09:29:43.666237 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0620 09:29:43.666267 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:44381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260620 09:29:43.666460 733 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 09:29:43.681865 736 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:29:43.682099 736 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:29:43.682127 736 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:29:43.683364 736 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:29:43.683406 736 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:29:43.683434 736 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.43.130
I20260620 09:29:43.685005 736 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:41053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.130.keytab
--principal=kudu/127.16.43.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.43.130:43087
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.43.130
--webserver_port=40973
--tserver_master_addrs=127.16.43.190:44381
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.736
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.43.130
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:29:43.685197 736 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:29:43.685467 736 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:29:43.691274 743 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:29:43.691463 744 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:29:43.691692 746 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:29:43.691833 736 server_base.cc:1034] running on GCE node
I20260620 09:29:43.692042 736 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:29:43.692292 736 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:29:43.693460 736 hybrid_clock.cc:648] HybridClock initialized: now 1781947783693433 us; error 42 us; skew 500 ppm
Jun 20 09:29:43 dist-test-slave-706q krb5kdc[31897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947783, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:29:43.696110 736 init.cc:377] Logged in from keytab as kudu/127.16.43.130@KRBTEST.COM (short username kudu)
I20260620 09:29:43.696580 736 webserver.cc:460] Webserver started at http://127.16.43.130:40973/ using document root <none> and password file <none>
I20260620 09:29:43.696767 736 fs_manager.cc:359] Metadata directory not provided
I20260620 09:29:43.696815 736 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:29:43.698109 736 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:29:43.698705 753 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:29:43.698899 736 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 09:29:43.698968 736 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-1/wal
uuid: "c87b67aa6f1a434c85643b5029ea8a91"
format_stamp: "Formatted at 2026-06-20 09:29:01 on dist-test-slave-706q"
server_key: "516ea6a6d812ad1e363f11cdfb7b7c73"
server_key_iv: "ca67562decf88946b3c07737280ab04c"
server_key_version: "encryptionkey@0"
I20260620 09:29:43.699327 736 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:29:43.713128 736 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:29:43.715258 736 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:29:43.715433 736 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:29:43.715705 736 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:29:43.716027 736 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:29:43.716060 736 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:29:43.716084 736 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:29:43.716115 736 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:29:43.722343 736 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.130:43087
I20260620 09:29:43.722440 866 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.130:43087 every 8 connection(s)
I20260620 09:29:43.722759 736 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-1/data/info.pb
I20260620 09:29:43.723928 869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:29:43.723520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:44381 (local address 127.16.43.130:36561)
0620 09:29:43.723702 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0620 09:29:43.723773 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:44381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":73,"threads_started":1}
W20260620 09:29:43.724344 867 heartbeater.cc:644] Failed to heartbeat to 127.16.43.190:44381 (0 consecutive failures): Network error: Failed to ping master at 127.16.43.190:44381: Client connection negotiation failed: client connection to 127.16.43.190:44381: connect: Connection refused (error 111)
I20260620 09:29:43.724654 869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:29:43.724566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:44381 (local address 127.16.43.130:43581)
0620 09:29:43.724611 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0620 09:29:43.724637 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:44381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260620 09:29:43.724903 869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:29:43.724853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:44381 (local address 127.16.43.130:38551)
0620 09:29:43.724874 (+ 21us) negotiation.cc:107] Waiting for socket to connect
0620 09:29:43.724892 (+ 18us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:44381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12}
W20260620 09:29:43.725025 867 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 09:29:43.729686 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 736
I20260620 09:29:43.730013 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.43.131:41523
--local_ip_for_outbound_sockets=127.16.43.131
--tserver_master_addrs=127.16.43.190:44381
--webserver_port=34609
--webserver_interface=127.16.43.131
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.131.keytab
--principal=kudu/127.16.43.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.43.148:41053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:29:43.816505 870 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:29:43.816756 870 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:29:43.816795 870 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:29:43.817965 870 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:29:43.818029 870 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:29:43.818073 870 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.43.131
I20260620 09:29:43.820257 870 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:41053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.131.keytab
--principal=kudu/127.16.43.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.43.131:41523
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.43.131
--webserver_port=34609
--tserver_master_addrs=127.16.43.190:44381
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.870
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.43.131
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:29:43.820542 870 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:29:43.820832 870 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:29:43.826449 875 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:29:43.826548 876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:29:43.826449 878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:29:43.826692 870 server_base.cc:1034] running on GCE node
I20260620 09:29:43.826941 870 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:29:43.827193 870 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:29:43.828392 870 hybrid_clock.cc:648] HybridClock initialized: now 1781947783828373 us; error 38 us; skew 500 ppm
Jun 20 09:29:43 dist-test-slave-706q krb5kdc[31897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947783, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:29:43.831238 870 init.cc:377] Logged in from keytab as kudu/127.16.43.131@KRBTEST.COM (short username kudu)
I20260620 09:29:43.831828 870 webserver.cc:460] Webserver started at http://127.16.43.131:34609/ using document root <none> and password file <none>
I20260620 09:29:43.832090 870 fs_manager.cc:359] Metadata directory not provided
I20260620 09:29:43.832167 870 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:29:43.833741 870 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 09:29:43.834585 885 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:29:43.834842 870 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:29:43.834973 870 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-2/wal
uuid: "804331f690c6404db86d9c664915e9d2"
format_stamp: "Formatted at 2026-06-20 09:29:02 on dist-test-slave-706q"
server_key: "76d950b760a461df80d6a522e8c6077e"
server_key_iv: "29895f931e63037c03a0a64c62b1fda6"
server_key_version: "encryptionkey@0"
I20260620 09:29:43.835409 870 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:29:43.843791 870 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:29:43.846342 870 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:29:43.846515 870 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:29:43.846761 870 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:29:43.847100 870 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:29:43.847159 870 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:29:43.847208 870 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:29:43.847242 870 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:29:43.853899 870 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.131:41523
I20260620 09:29:43.853945 998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.131:41523 every 8 connection(s)
I20260620 09:29:43.854321 870 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781947562949448-16558-0/minicluster-data/ts-2/data/info.pb
I20260620 09:29:43.855510 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 870
I20260620 09:29:43.855612 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:29:43.855116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:44381 (local address 127.16.43.131:36975)
0620 09:29:43.855335 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0620 09:29:43.855411 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:44381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":200,"threads_started":1}
W20260620 09:29:43.856074 999 heartbeater.cc:644] Failed to heartbeat to 127.16.43.190:44381 (0 consecutive failures): Network error: Failed to ping master at 127.16.43.190:44381: Client connection negotiation failed: client connection to 127.16.43.190:44381: connect: Connection refused (error 111)
I20260620 09:29:43.856384 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:29:43.856301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:44381 (local address 127.16.43.131:56787)
0620 09:29:43.856335 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0620 09:29:43.856363 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:44381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20260620 09:29:43.856659 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:29:43.856606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:44381 (local address 127.16.43.131:40745)
0620 09:29:43.856627 (+ 21us) negotiation.cc:107] Waiting for socket to connect
0620 09:29:43.856646 (+ 19us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:44381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12}
W20260620 09:29:43.856793 999 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 09:29:44.667091 1003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:29:44.666838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:44381 (local address 127.16.43.129:55471)
0620 09:29:44.666982 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 09:29:44.667026 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:44381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":64,"threads_started":1}
I20260620 09:29:44.725750 1004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:29:44.725506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:44381 (local address 127.16.43.130:55507)
0620 09:29:44.725651 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 09:29:44.725695 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:44381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":59,"threads_started":1}
I20260620 09:29:44.729749 1005 raft_consensus.cc:482] T b242f47f6b714327bd03c0dc3b32c374 P f4e2727ee8d7413a9aa64423a656905a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:29:44.729911 1005 raft_consensus.cc:504] T b242f47f6b714327bd03c0dc3b32c374 P f4e2727ee8d7413a9aa64423a656905a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "804331f690c6404db86d9c664915e9d2" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 41523 } } peers { permanent_uuid: "f4e2727ee8d7413a9aa64423a656905a" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 42439 } } peers { permanent_uuid: "c87b67aa6f1a434c85643b5029ea8a91" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 43087 } }
I20260620 09:29:44.730237 1005 leader_election.cc:288] T b242f47f6b714327bd03c0dc3b32c374 P f4e2727ee8d7413a9aa64423a656905a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 804331f690c6404db86d9c664915e9d2 (127.16.43.131:41523), c87b67aa6f1a434c85643b5029ea8a91 (127.16.43.130:43087)
Jun 20 09:29:44 dist-test-slave-706q krb5kdc[31897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947783, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.129@KRBTEST.COM for kudu/127.16.43.131@KRBTEST.COM
Jun 20 09:29:44 dist-test-slave-706q krb5kdc[31897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947783, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.129@KRBTEST.COM for kudu/127.16.43.130@KRBTEST.COM
I20260620 09:29:44.744585 1008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:29:44.730547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.129:60991 (local address 127.16.43.131:41523)
0620 09:29:44.730728 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 09:29:44.730739 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 09:29:44.730771 (+ 32us) server_negotiation.cc:407] Connection header received
0620 09:29:44.731190 (+ 419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:29:44.731192 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:29:44.731249 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:29:44.731405 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:29:44.732202 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:29:44.733107 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:29:44.734123 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:29:44.734536 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:29:44.734680 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:29:44.740896 (+ 6216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:29:44.740901 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:29:44.740901 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:29:44.740918 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:29:44.742816 (+ 1898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:29:44.743309 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:29:44.743313 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:29:44.743313 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:29:44.743417 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:29:44.743698 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:29:44.743699 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:29:44.743699 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:29:44.743969 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:29:44.744063 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 09:29:44.744350 (+ 287us) server_negotiation.cc:299] Negotiation successful
0620 09:29:44.744476 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":75,"threads_started":1}
I20260620 09:29:44.744585 1006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:29:44.730423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.131:41523 (local address 127.16.43.129:60991)
0620 09:29:44.730639 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0620 09:29:44.730648 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 09:29:44.730684 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:29:44.731460 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:29:44.731463 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:29:44.731885 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:29:44.732079 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:29:44.732084 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:29:44.733222 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:29:44.733223 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:29:44.734034 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:29:44.734036 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:29:44.734644 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:29:44.734645 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:29:44.734761 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:29:44.738996 (+ 4235us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:29:44.739006 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:29:44.740755 (+ 1749us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:29:44.742934 (+ 2179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:29:44.742941 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:29:44.742943 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:29:44.743223 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:29:44.743488 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:29:44.743489 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:29:44.743489 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:29:44.743630 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:29:44.744044 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:29:44.744050 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:29:44.744184 (+ 134us) client_negotiation.cc:769] Sending connection context
0620 09:29:44.744284 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 09:29:44.744487 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"mutex_wait_us":18,"thread_start_us":55,"threads_started":1}
I20260620 09:29:44.744630 1007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:29:44.730558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.129:38051 (local address 127.16.43.130:43087)
0620 09:29:44.730737 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 09:29:44.730745 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 09:29:44.730772 (+ 27us) server_negotiation.cc:407] Connection header received
0620 09:29:44.731218 (+ 446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:29:44.731221 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:29:44.731279 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:29:44.731441 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:29:44.732309 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:29:44.733138 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:29:44.734147 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:29:44.734546 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:29:44.734680 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:29:44.741286 (+ 6606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:29:44.741290 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:29:44.741290 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:29:44.741301 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:29:44.742955 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:29:44.743360 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:29:44.743363 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:29:44.743364 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:29:44.743461 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:29:44.743721 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:29:44.743722 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:29:44.743722 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:29:44.743965 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:29:44.744059 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 09:29:44.744357 (+ 298us) server_negotiation.cc:299] Negotiation successful
0620 09:29:44.744511 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":68,"threads_started":1}
I20260620 09:29:44.744928 1003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:29:44.730405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.130:43087 (local address 127.16.43.129:38051)
0620 09:29:44.730482 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0620 09:29:44.730493 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 09:29:44.730553 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:29:44.731460 (+ 907us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:29:44.731463 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:29:44.731882 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:29:44.732079 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:29:44.732085 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:29:44.733256 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:29:44.733257 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:29:44.734035 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:29:44.734037 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:29:44.734651 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:29:44.734652 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:29:44.734763 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:29:44.738992 (+ 4229us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:29:44.739007 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:29:44.741143 (+ 2136us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:29:44.743068 (+ 1925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:29:44.743069 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:29:44.743070 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:29:44.743270 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:29:44.743569 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:29:44.743570 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:29:44.743571 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:29:44.743650 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:29:44.744038 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:29:44.744045 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:29:44.744184 (+ 139us) client_negotiation.cc:769] Sending connection context
0620 09:29:44.744290 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 09:29:44.744568 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52,"mutex_wait_us":208}
W20260620 09:29:44.745512 620 leader_election.cc:341] T b242f47f6b714327bd03c0dc3b32c374 P f4e2727ee8d7413a9aa64423a656905a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 804331f690c6404db86d9c664915e9d2 (127.16.43.131:41523): Not found: Tablet not found: b242f47f6b714327bd03c0dc3b32c374
W20260620 09:29:44.745599 622 leader_election.cc:341] T b242f47f6b714327bd03c0dc3b32c374 P f4e2727ee8d7413a9aa64423a656905a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c87b67aa6f1a434c85643b5029ea8a91 (127.16.43.130:43087): Not found: Tablet not found: b242f47f6b714327bd03c0dc3b32c374
I20260620 09:29:44.745671 622 leader_election.cc:302] T b242f47f6b714327bd03c0dc3b32c374 P f4e2727ee8d7413a9aa64423a656905a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f4e2727ee8d7413a9aa64423a656905a; no voters: 804331f690c6404db86d9c664915e9d2, c87b67aa6f1a434c85643b5029ea8a91
I20260620 09:29:44.745791 1005 raft_consensus.cc:2726] T b242f47f6b714327bd03c0dc3b32c374 P f4e2727ee8d7413a9aa64423a656905a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 09:29:44.857322 1009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:29:44.857114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:44381 (local address 127.16.43.131:51505)
0620 09:29:44.857226 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0620 09:29:44.857274 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:44381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":54,"threads_started":1}
I20260620 09:29:45.668090 1010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:29:45.667819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:44381 (local address 127.16.43.129:52285)
0620 09:29:45.667973 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 09:29:45.668031 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:44381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":75,"threads_started":1}
I20260620 09:29:45.726449 1011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:29:45.726223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:44381 (local address 127.16.43.130:41939)
0620 09:29:45.726365 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 09:29:45.726402 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:44381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":65,"threads_started":1}
I20260620 09:29:45.857997 1012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:29:45.857764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:44381 (local address 127.16.43.131:34209)
0620 09:29:45.857913 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 09:29:45.857955 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:44381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":60,"threads_started":1}
I20260620 09:29:46.668874 1013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:29:46.668643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:44381 (local address 127.16.43.129:40397)
0620 09:29:46.668773 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0620 09:29:46.668822 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:44381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":67,"threads_started":1}
I20260620 09:29:46.727283 1014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:29:46.727019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:44381 (local address 127.16.43.130:40495)
0620 09:29:46.727188 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 09:29:46.727236 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:44381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":72,"threads_started":1}
I20260620 09:29:46.767735 1015 raft_consensus.cc:482] T b242f47f6b714327bd03c0dc3b32c374 P f4e2727ee8d7413a9aa64423a656905a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:29:46.767886 1015 raft_consensus.cc:504] T b242f47f6b714327bd03c0dc3b32c374 P f4e2727ee8d7413a9aa64423a656905a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "804331f690c6404db86d9c664915e9d2" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 41523 } } peers { permanent_uuid: "f4e2727ee8d7413a9aa64423a656905a" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 42439 } } peers { permanent_uuid: "c87b67aa6f1a434c85643b5029ea8a91" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 43087 } }
I20260620 09:29:46.768090 1015 leader_election.cc:288] T b242f47f6b714327bd03c0dc3b32c374 P f4e2727ee8d7413a9aa64423a656905a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 804331f690c6404db86d9c664915e9d2 (127.16.43.131:41523), c87b67aa6f1a434c85643b5029ea8a91 (127.16.43.130:43087)
W20260620 09:29:46.768635 620 leader_election.cc:341] T b242f47f6b714327bd03c0dc3b32c374 P f4e2727ee8d7413a9aa64423a656905a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 804331f690c6404db86d9c664915e9d2 (127.16.43.131:41523): Not found: Tablet not found: b242f47f6b714327bd03c0dc3b32c374
W20260620 09:29:46.768710 622 leader_election.cc:341] T b242f47f6b714327bd03c0dc3b32c374 P f4e2727ee8d7413a9aa64423a656905a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c87b67aa6f1a434c85643b5029ea8a91 (127.16.43.130:43087): Not found: Tablet not found: b242f47f6b714327bd03c0dc3b32c374
I20260620 09:29:46.768744 622 leader_election.cc:302] T b242f47f6b714327bd03c0dc3b32c374 P f4e2727ee8d7413a9aa64423a656905a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f4e2727ee8d7413a9aa64423a656905a; no voters: 804331f690c6404db86d9c664915e9d2, c87b67aa6f1a434c85643b5029ea8a91
I20260620 09:29:46.768836 1015 raft_consensus.cc:2726] T b242f47f6b714327bd03c0dc3b32c374 P f4e2727ee8d7413a9aa64423a656905a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 09:29:46.858878 1016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:29:46.858645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:44381 (local address 127.16.43.131:36873)
0620 09:29:46.858790 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 09:29:46.858832 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:44381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":77,"threads_started":1}
I20260620 09:29:47.669615 1017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:29:47.669375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:44381 (local address 127.16.43.129:56549)
0620 09:29:47.669525 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 09:29:47.669565 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:44381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":62,"threads_started":1}
I20260620 09:29:47.728034 1018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:29:47.727800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:44381 (local address 127.16.43.130:50639)
0620 09:29:47.727931 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0620 09:29:47.727987 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:44381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":59,"threads_started":1}
I20260620 09:29:47.859728 1019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:29:47.859428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:44381 (local address 127.16.43.131:41663)
0620 09:29:47.859639 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0620 09:29:47.859685 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:44381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":66,"threads_started":1}
I20260620 09:29:48.670487 1020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:29:48.670222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:44381 (local address 127.16.43.129:40977)
0620 09:29:48.670378 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 09:29:48.670434 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:44381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":63,"threads_started":1}
I20260620 09:29:48.728864 1021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:29:48.728579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:44381 (local address 127.16.43.130:37423)
0620 09:29:48.728751 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0620 09:29:48.728813 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:44381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":66,"threads_started":1}
I20260620 09:29:48.860497 1022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:29:48.860289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:44381 (local address 127.16.43.131:46919)
0620 09:29:48.860416 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0620 09:29:48.860451 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:44381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":59,"threads_started":1}
I20260620 09:29:49.671465 1023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:29:49.671140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:44381 (local address 127.16.43.129:34307)
0620 09:29:49.671297 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 09:29:49.671390 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:44381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":63,"threads_started":1}
I20260620 09:29:49.729642 1024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:29:49.729401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:44381 (local address 127.16.43.130:46623)
0620 09:29:49.729547 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 09:29:49.729595 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:44381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":63,"threads_started":1}
I20260620 09:29:49.739848 1025 raft_consensus.cc:482] T b242f47f6b714327bd03c0dc3b32c374 P f4e2727ee8d7413a9aa64423a656905a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:29:49.739950 1025 raft_consensus.cc:504] T b242f47f6b714327bd03c0dc3b32c374 P f4e2727ee8d7413a9aa64423a656905a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "804331f690c6404db86d9c664915e9d2" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 41523 } } peers { permanent_uuid: "f4e2727ee8d7413a9aa64423a656905a" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 42439 } } peers { permanent_uuid: "c87b67aa6f1a434c85643b5029ea8a91" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 43087 } }
I20260620 09:29:49.740185 1025 leader_election.cc:288] T b242f47f6b714327bd03c0dc3b32c374 P f4e2727ee8d7413a9aa64423a656905a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 804331f690c6404db86d9c664915e9d2 (127.16.43.131:41523), c87b67aa6f1a434c85643b5029ea8a91 (127.16.43.130:43087)
W20260620 09:29:49.740739 622 leader_election.cc:341] T b242f47f6b714327bd03c0dc3b32c374 P f4e2727ee8d7413a9aa64423a656905a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c87b67aa6f1a434c85643b5029ea8a91 (127.16.43.130:43087): Not found: Tablet not found: b242f47f6b714327bd03c0dc3b32c374
W20260620 09:29:49.740865 620 leader_election.cc:341] T b242f47f6b714327bd03c0dc3b32c374 P f4e2727ee8d7413a9aa64423a656905a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 804331f690c6404db86d9c664915e9d2 (127.16.43.131:41523): Not found: Tablet not found: b242f47f6b714327bd03c0dc3b32c374
I20260620 09:29:49.740942 620 leader_election.cc:302] T b242f47f6b714327bd03c0dc3b32c374 P f4e2727ee8d7413a9aa64423a656905a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f4e2727ee8d7413a9aa64423a656905a; no voters: 804331f690c6404db86d9c664915e9d2, c87b67aa6f1a434c85643b5029ea8a91
I20260620 09:29:49.741039 1025 raft_consensus.cc:2726] T b242f47f6b714327bd03c0dc3b32c374 P f4e2727ee8d7413a9aa64423a656905a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 09:29:49.861243 1026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:29:49.861019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:44381 (local address 127.16.43.131:41589)
0620 09:29:49.861152 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 09:29:49.861193 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:44381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":72,"threads_started":1}
I20260620 09:29:50.672281 1027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:29:50.672029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:44381 (local address 127.16.43.129:54293)
0620 09:29:50.672183 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 09:29:50.672230 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:44381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":67,"threads_started":1}
I20260620 09:29:50.730446 1028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:29:50.730179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:44381 (local address 127.16.43.130:33361)
0620 09:29:50.730346 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 09:29:50.730401 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:44381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":61,"threads_started":1}
I20260620 09:29:50.862074 1029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:29:50.861806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:44381 (local address 127.16.43.131:33811)
0620 09:29:50.861966 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 09:29:50.862018 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:44381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":100,"threads_started":1}
I20260620 09:29:51.673274 1030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:29:51.672977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:44381 (local address 127.16.43.129:49809)
0620 09:29:51.673171 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0620 09:29:51.673219 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:44381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":100,"threads_started":1}
I20260620 09:29:51.731369 1031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:29:51.731072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:44381 (local address 127.16.43.130:50799)
0620 09:29:51.731255 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0620 09:29:51.731313 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:44381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":82,"threads_started":1}
I20260620 09:29:51.862906 1032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:29:51.862676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:44381 (local address 127.16.43.131:59263)
0620 09:29:51.862821 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 09:29:51.862860 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:44381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":86,"threads_started":1}
I20260620 09:29:52.674193 1033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:29:52.673916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:44381 (local address 127.16.43.129:40505)
0620 09:29:52.674078 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 09:29:52.674139 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:44381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":76,"threads_started":1}
I20260620 09:29:52.732195 1034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:29:52.731898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:44381 (local address 127.16.43.130:51363)
0620 09:29:52.732095 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0620 09:29:52.732143 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:44381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":77,"threads_started":1}
I20260620 09:29:52.816684 1035 raft_consensus.cc:482] T b242f47f6b714327bd03c0dc3b32c374 P f4e2727ee8d7413a9aa64423a656905a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:29:52.816793 1035 raft_consensus.cc:504] T b242f47f6b714327bd03c0dc3b32c374 P f4e2727ee8d7413a9aa64423a656905a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "804331f690c6404db86d9c664915e9d2" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 41523 } } peers { permanent_uuid: "f4e2727ee8d7413a9aa64423a656905a" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 42439 } } peers { permanent_uuid: "c87b67aa6f1a434c85643b5029ea8a91" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 43087 } }
I20260620 09:29:52.816931 1035 leader_election.cc:288] T b242f47f6b714327bd03c0dc3b32c374 P f4e2727ee8d7413a9aa64423a656905a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 804331f690c6404db86d9c664915e9d2 (127.16.43.131:41523), c87b67aa6f1a434c85643b5029ea8a91 (127.16.43.130:43087)
W20260620 09:29:52.817389 622 leader_election.cc:341] T b242f47f6b714327bd03c0dc3b32c374 P f4e2727ee8d7413a9aa64423a656905a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c87b67aa6f1a434c85643b5029ea8a91 (127.16.43.130:43087): Not found: Tablet not found: b242f47f6b714327bd03c0dc3b32c374
W20260620 09:29:52.817466 620 leader_election.cc:341] T b242f47f6b714327bd03c0dc3b32c374 P f4e2727ee8d7413a9aa64423a656905a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 804331f690c6404db86d9c664915e9d2 (127.16.43.131:41523): Not found: Tablet not found: b242f47f6b714327bd03c0dc3b32c374
I20260620 09:29:52.817543 620 leader_election.cc:302] T b242f47f6b714327bd03c0dc3b32c374 P f4e2727ee8d7413a9aa64423a656905a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f4e2727ee8d7413a9aa64423a656905a; no voters: 804331f690c6404db86d9c664915e9d2, c87b67aa6f1a434c85643b5029ea8a91
I20260620 09:29:52.817632 1035 raft_consensus.cc:2726] T b242f47f6b714327bd03c0dc3b32c374 P f4e2727ee8d7413a9aa64423a656905a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 09:29:52.863785 1036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:29:52.863460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:44381 (local address 127.16.43.131:38241)
0620 09:29:52.863675 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0620 09:29:52.863718 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:44381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":136,"threads_started":1}
I20260620 09:29:53.675058 1037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:29:53.674823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:44381 (local address 127.16.43.129:39887)
0620 09:29:53.674977 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 09:29:53.675011 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:44381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":94,"threads_started":1}
I20260620 09:29:53.733019 1038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:29:53.732778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:44381 (local address 127.16.43.130:49715)
0620 09:29:53.732925 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 09:29:53.732972 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:44381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":63,"threads_started":1}
I20260620 09:29:53.864599 1039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:29:53.864361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:44381 (local address 127.16.43.131:45935)
0620 09:29:53.864506 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 09:29:53.864548 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:44381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":69,"threads_started":1}
I20260620 09:29:54.675841 1040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:29:54.675571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:44381 (local address 127.16.43.129:43499)
0620 09:29:54.675745 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 09:29:54.675783 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:44381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":73,"threads_started":1}
I20260620 09:29:54.733882 1041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:29:54.733632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:44381 (local address 127.16.43.130:35389)
0620 09:29:54.733785 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 09:29:54.733833 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:44381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":85,"threads_started":1}
I20260620 09:29:54.865646 1042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:29:54.865240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:44381 (local address 127.16.43.131:47681)
0620 09:29:54.865501 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0620 09:29:54.865568 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:44381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":139,"threads_started":1}
I20260620 09:29:55.676694 1043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:29:55.676357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:44381 (local address 127.16.43.129:57783)
0620 09:29:55.676560 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0620 09:29:55.676609 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:44381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":79,"threads_started":1}
I20260620 09:29:55.734826 1044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:29:55.734479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:44381 (local address 127.16.43.130:55641)
0620 09:29:55.734658 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 09:29:55.734758 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:44381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":77,"threads_started":1}
I20260620 09:29:55.866549 1045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:29:55.866219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:44381 (local address 127.16.43.131:42669)
0620 09:29:55.866392 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 09:29:55.866495 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:44381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":80,"threads_started":1}
I20260620 09:29:56.172319 1046 raft_consensus.cc:482] T b242f47f6b714327bd03c0dc3b32c374 P f4e2727ee8d7413a9aa64423a656905a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:29:56.172413 1046 raft_consensus.cc:504] T b242f47f6b714327bd03c0dc3b32c374 P f4e2727ee8d7413a9aa64423a656905a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "804331f690c6404db86d9c664915e9d2" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 41523 } } peers { permanent_uuid: "f4e2727ee8d7413a9aa64423a656905a" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 42439 } } peers { permanent_uuid: "c87b67aa6f1a434c85643b5029ea8a91" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 43087 } }
I20260620 09:29:56.172647 1046 leader_election.cc:288] T b242f47f6b714327bd03c0dc3b32c374 P f4e2727ee8d7413a9aa64423a656905a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 804331f690c6404db86d9c664915e9d2 (127.16.43.131:41523), c87b67aa6f1a434c85643b5029ea8a91 (127.16.43.130:43087)
W20260620 09:29:56.173118 622 leader_election.cc:341] T b242f47f6b714327bd03c0dc3b32c374 P f4e2727ee8d7413a9aa64423a656905a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c87b67aa6f1a434c85643b5029ea8a91 (127.16.43.130:43087): Not found: Tablet not found: b242f47f6b714327bd03c0dc3b32c374
W20260620 09:29:56.173205 620 leader_election.cc:341] T b242f47f6b714327bd03c0dc3b32c374 P f4e2727ee8d7413a9aa64423a656905a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 804331f690c6404db86d9c664915e9d2 (127.16.43.131:41523): Not found: Tablet not found: b242f47f6b714327bd03c0dc3b32c374
I20260620 09:29:56.173237 620 leader_election.cc:302] T b242f47f6b714327bd03c0dc3b32c374 P f4e2727ee8d7413a9aa64423a656905a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f4e2727ee8d7413a9aa64423a656905a; no voters: 804331f690c6404db86d9c664915e9d2, c87b67aa6f1a434c85643b5029ea8a91
I20260620 09:29:56.173331 1046 raft_consensus.cc:2726] T b242f47f6b714327bd03c0dc3b32c374 P f4e2727ee8d7413a9aa64423a656905a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 09:29:56.677580 1047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:29:56.677325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:44381 (local address 127.16.43.129:48229)
0620 09:29:56.677482 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 09:29:56.677528 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:44381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":64,"threads_started":1}
I20260620 09:29:56.735783 1048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:29:56.735471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:44381 (local address 127.16.43.130:44493)
0620 09:29:56.735684 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0620 09:29:56.735731 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:44381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":132,"threads_started":1}
I20260620 09:29:56.867228 1049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:29:56.866980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:44381 (local address 127.16.43.131:38915)
0620 09:29:56.867127 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 09:29:56.867170 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:44381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":69,"threads_started":1}
I20260620 09:29:57.678447 1050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:29:57.678161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:44381 (local address 127.16.43.129:48689)
0620 09:29:57.678347 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0620 09:29:57.678394 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:44381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":88,"threads_started":1}
I20260620 09:29:57.736610 1051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:29:57.736394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:44381 (local address 127.16.43.130:56379)
0620 09:29:57.736524 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0620 09:29:57.736564 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:44381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":64,"threads_started":1}
I20260620 09:29:57.868103 1052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:29:57.867843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:44381 (local address 127.16.43.131:54037)
0620 09:29:57.867998 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 09:29:57.868051 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:44381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":63,"threads_started":1}
I20260620 09:29:58.679649 1053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:29:58.679214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:44381 (local address 127.16.43.129:34561)
0620 09:29:58.679459 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0620 09:29:58.679508 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:44381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":111,"threads_started":1}
I20260620 09:29:58.737504 1054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:29:58.737228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:44381 (local address 127.16.43.130:48973)
0620 09:29:58.737383 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 09:29:58.737427 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:44381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":63,"threads_started":1}
I20260620 09:29:58.860107 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 572
I20260620 09:29:58.864849 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 736
I20260620 09:29:58.868896 1055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:29:58.868685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:44381 (local address 127.16.43.131:39369)
0620 09:29:58.868810 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0620 09:29:58.868851 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.190:44381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":77,"threads_started":1}
I20260620 09:29:58.869158 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 870
2026-06-20T09:29:58Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58661 ms)
[----------] 30 tests from SecurityITest (235928 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781947562949448-16558-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:29:58 dist-test-slave-706q krb5kdc[1058](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:29:58 dist-test-slave-706q krb5kdc[1058](info): set up 2 sockets
Jun 20 09:29:58 dist-test-slave-706q krb5kdc[1058](info): commencing operation
krb5kdc: starting...
W20260620 09:29:59.920353 16558 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781947562949448-16558-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781947562949448-16558-0/krb5kdc/test-user.keytab.
Jun 20 09:30:00 dist-test-slave-706q krb5kdc[1058](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947800, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:30:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:30:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.43.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.43.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.43.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.190.keytab.
Entry for principal kudu/127.16.43.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.190.keytab.
I20260620 09:30:00.055624 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781947562949448-16558-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781947562949448-16558-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781947562949448-16558-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.43.190:44157
--webserver_interface=127.16.43.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.190.keytab
--principal=kudu/127.16.43.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.43.148:35629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.43.190:44157
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781947562949448-16558-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781947562949448-16558-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781947562949448-16558-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:30:00.140822 1071 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:30:00.141076 1071 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:30:00.141114 1071 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:30:00.142277 1071 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:30:00.142354 1071 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:30:00.142378 1071 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:30:00.142400 1071 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:30:00.142427 1071 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:30:00.143967 1071 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:35629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781947562949448-16558-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781947562949448-16558-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.43.190:44157
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.190.keytab
--principal=kudu/127.16.43.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.43.190:44157
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.43.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1071
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781947562949448-16558-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:30:00.144240 1071 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:30:00.144477 1071 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:30:00.149993 1077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:30:00.150068 1076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:30:00.150147 1071 server_base.cc:1034] running on GCE node
W20260620 09:30:00.149993 1079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:30:00.150578 1071 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:30:00.150925 1071 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:30:00.152107 1071 hybrid_clock.cc:648] HybridClock initialized: now 1781947800152091 us; error 45 us; skew 500 ppm
Jun 20 09:30:00 dist-test-slave-706q krb5kdc[1058](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947800, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:30:00.154915 1071 init.cc:377] Logged in from keytab as kudu/127.16.43.190@KRBTEST.COM (short username kudu)
I20260620 09:30:00.155392 1071 webserver.cc:460] Webserver started at http://127.16.43.190:36633/ using document root <none> and password file <none>
I20260620 09:30:00.155678 1071 fs_manager.cc:359] Metadata directory not provided
I20260620 09:30:00.155750 1071 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:30:00.155905 1071 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:30:00.156992 1071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781947562949448-16558-0/minicluster-data/master-0/data/instance:
uuid: "04549979371e455e953c09ec661ccecc"
format_stamp: "Formatted at 2026-06-20 09:30:00 on dist-test-slave-706q"
server_key: "6ab606948c9a1ee28dc8b451209b0fc1"
server_key_iv: "b72215dff61c074ff1baef009677e0b6"
server_key_version: "encryptionkey@0"
I20260620 09:30:00.157377 1071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781947562949448-16558-0/minicluster-data/master-0/wal/instance:
uuid: "04549979371e455e953c09ec661ccecc"
format_stamp: "Formatted at 2026-06-20 09:30:00 on dist-test-slave-706q"
server_key: "6ab606948c9a1ee28dc8b451209b0fc1"
server_key_iv: "b72215dff61c074ff1baef009677e0b6"
server_key_version: "encryptionkey@0"
I20260620 09:30:00.158833 1071 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 09:30:00.159651 1086 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:30:00.159896 1071 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:30:00.159998 1071 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781947562949448-16558-0/minicluster-data/master-0/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781947562949448-16558-0/minicluster-data/master-0/wal
uuid: "04549979371e455e953c09ec661ccecc"
format_stamp: "Formatted at 2026-06-20 09:30:00 on dist-test-slave-706q"
server_key: "6ab606948c9a1ee28dc8b451209b0fc1"
server_key_iv: "b72215dff61c074ff1baef009677e0b6"
server_key_version: "encryptionkey@0"
I20260620 09:30:00.160076 1071 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781947562949448-16558-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781947562949448-16558-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781947562949448-16558-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:30:00.168985 1071 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:30:00.170856 1071 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:30:00.171010 1071 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:30:00.175074 1071 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.190:44157
I20260620 09:30:00.175151 1138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.190:44157 every 8 connection(s)
I20260620 09:30:00.175487 1071 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
I20260620 09:30:00.176461 1139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:30:00.178918 1139 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04549979371e455e953c09ec661ccecc: Bootstrap starting.
I20260620 09:30:00.179494 1139 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 04549979371e455e953c09ec661ccecc: Neither blocks nor log segments found. Creating new log.
I20260620 09:30:00.179860 1139 log.cc:824] T 00000000000000000000000000000000 P 04549979371e455e953c09ec661ccecc: Log is configured to *not* fsync() on all Append() calls
I20260620 09:30:00.180577 1139 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04549979371e455e953c09ec661ccecc: No bootstrap required, opened a new log
I20260620 09:30:00.180928 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 1071
I20260620 09:30:00.181037 16558 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781947562949448-16558-0/minicluster-data/master-0/wal/instance
I20260620 09:30:00.181247 16558 external_mini_cluster.cc:1428] Setting key 409c2cbea6b034c8a7e29e7b0ab125eb
I20260620 09:30:00.182809 1139 raft_consensus.cc:348] T 00000000000000000000000000000000 P 04549979371e455e953c09ec661ccecc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04549979371e455e953c09ec661ccecc" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 44157 } }
I20260620 09:30:00.182947 1139 raft_consensus.cc:374] T 00000000000000000000000000000000 P 04549979371e455e953c09ec661ccecc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:30:00.182971 1139 raft_consensus.cc:729] T 00000000000000000000000000000000 P 04549979371e455e953c09ec661ccecc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04549979371e455e953c09ec661ccecc, State: Initialized, Role: FOLLOWER
I20260620 09:30:00.183153 1139 consensus_queue.cc:260] T 00000000000000000000000000000000 P 04549979371e455e953c09ec661ccecc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04549979371e455e953c09ec661ccecc" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 44157 } }
I20260620 09:30:00.183250 1139 raft_consensus.cc:388] T 00000000000000000000000000000000 P 04549979371e455e953c09ec661ccecc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:30:00.183285 1139 raft_consensus.cc:482] T 00000000000000000000000000000000 P 04549979371e455e953c09ec661ccecc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:30:00.183341 1139 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 04549979371e455e953c09ec661ccecc [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:30:00.184448 1139 raft_consensus.cc:504] T 00000000000000000000000000000000 P 04549979371e455e953c09ec661ccecc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04549979371e455e953c09ec661ccecc" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 44157 } }
I20260620 09:30:00.184540 1139 leader_election.cc:302] T 00000000000000000000000000000000 P 04549979371e455e953c09ec661ccecc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 04549979371e455e953c09ec661ccecc; no voters:
I20260620 09:30:00.184736 1139 leader_election.cc:288] T 00000000000000000000000000000000 P 04549979371e455e953c09ec661ccecc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:30:00.184785 1144 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 04549979371e455e953c09ec661ccecc [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:30:00.184957 1144 raft_consensus.cc:686] T 00000000000000000000000000000000 P 04549979371e455e953c09ec661ccecc [term 1 LEADER]: Becoming Leader. State: Replica: 04549979371e455e953c09ec661ccecc, State: Running, Role: LEADER
I20260620 09:30:00.185017 1139 sys_catalog.cc:564] T 00000000000000000000000000000000 P 04549979371e455e953c09ec661ccecc [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:30:00.185138 1144 consensus_queue.cc:237] T 00000000000000000000000000000000 P 04549979371e455e953c09ec661ccecc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04549979371e455e953c09ec661ccecc" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 44157 } }
I20260620 09:30:00.186118 1145 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04549979371e455e953c09ec661ccecc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "04549979371e455e953c09ec661ccecc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04549979371e455e953c09ec661ccecc" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 44157 } } }
I20260620 09:30:00.186205 1145 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04549979371e455e953c09ec661ccecc [sys.catalog]: This master's current role is: LEADER
I20260620 09:30:00.186549 1157 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:30:00.186558 1146 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04549979371e455e953c09ec661ccecc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 04549979371e455e953c09ec661ccecc. Latest consensus state: current_term: 1 leader_uuid: "04549979371e455e953c09ec661ccecc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04549979371e455e953c09ec661ccecc" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 44157 } } }
I20260620 09:30:00.186866 1146 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04549979371e455e953c09ec661ccecc [sys.catalog]: This master's current role is: LEADER
I20260620 09:30:00.187340 1157 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:30:00.189239 1157 catalog_manager.cc:1347] Generated new cluster ID: be807b0e135940bda1e67e4baeac3ec5
I20260620 09:30:00.189291 1157 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 09:30:00 dist-test-slave-706q krb5kdc[1058](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947800, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.43.190@KRBTEST.COM
I20260620 09:30:00.194132 1143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:00.181787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38518 (local address 127.16.43.190:44157)
0620 09:30:00.181988 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:00.181997 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:00.182023 (+ 26us) server_negotiation.cc:407] Connection header received
0620 09:30:00.182812 (+ 789us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:00.182820 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:00.182874 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:00.183062 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:30:00.183932 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:00.185098 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:00.186272 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:00.186948 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:00.187060 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:00.190095 (+ 3035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:30:00.190100 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:30:00.190100 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:30:00.190112 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:30:00.192135 (+ 2023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:30:00.192533 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:30:00.192536 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:30:00.192537 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:30:00.192682 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:30:00.192893 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:30:00.192894 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:30:00.192895 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:30:00.193225 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:30:00.193403 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:00.193472 (+ 69us) server_negotiation.cc:299] Negotiation successful
0620 09:30:00.193744 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":102,"threads_started":1}
I20260620 09:30:00.220989 1157 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:30:00.221778 1157 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:30:00.241160 1157 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 04549979371e455e953c09ec661ccecc: Generated new TSK 0
I20260620 09:30:00.241405 1157 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.43.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.43.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.43.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.129.keytab.
Entry for principal kudu/127.16.43.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.129.keytab.
I20260620 09:30:00.266875 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781947562949448-16558-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781947562949448-16558-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781947562949448-16558-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781947562949448-16558-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.43.129:0
--local_ip_for_outbound_sockets=127.16.43.129
--webserver_interface=127.16.43.129
--webserver_port=0
--tserver_master_addrs=127.16.43.190:44157
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.129.keytab
--principal=kudu/127.16.43.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.43.148:35629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781947562949448-16558-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781947562949448-16558-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781947562949448-16558-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:30:00.350054 1165 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:30:00.350277 1165 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:30:00.350302 1165 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:30:00.351434 1165 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:30:00.351471 1165 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:30:00.351497 1165 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.43.129
I20260620 09:30:00.353027 1165 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:35629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781947562949448-16558-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781947562949448-16558-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.129.keytab
--principal=kudu/127.16.43.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.43.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781947562949448-16558-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.43.129
--webserver_port=0
--tserver_master_addrs=127.16.43.190:44157
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1165
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.43.129
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781947562949448-16558-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:30:00.353235 1165 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:30:00.353416 1165 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:30:00.358711 1171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:30:00.358726 1173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:30:00.358760 1165 server_base.cc:1034] running on GCE node
W20260620 09:30:00.358759 1170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:30:00.359177 1165 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:30:00.359449 1165 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:30:00.360644 1165 hybrid_clock.cc:648] HybridClock initialized: now 1781947800360641 us; error 61 us; skew 500 ppm
Jun 20 09:30:00 dist-test-slave-706q krb5kdc[1058](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947800, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:30:00.363410 1165 init.cc:377] Logged in from keytab as kudu/127.16.43.129@KRBTEST.COM (short username kudu)
I20260620 09:30:00.363919 1165 webserver.cc:460] Webserver started at http://127.16.43.129:41121/ using document root <none> and password file <none>
I20260620 09:30:00.364142 1165 fs_manager.cc:359] Metadata directory not provided
I20260620 09:30:00.364185 1165 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:30:00.364276 1165 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:30:00.365298 1165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781947562949448-16558-0/minicluster-data/ts-0/data/instance:
uuid: "5537713527764d9590472593f39e4f02"
format_stamp: "Formatted at 2026-06-20 09:30:00 on dist-test-slave-706q"
server_key: "023fce16c9573930853adae1b1a21585"
server_key_iv: "4ece6191943c70c5e27053447ac5182f"
server_key_version: "encryptionkey@0"
I20260620 09:30:00.365600 1165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781947562949448-16558-0/minicluster-data/ts-0/wal/instance:
uuid: "5537713527764d9590472593f39e4f02"
format_stamp: "Formatted at 2026-06-20 09:30:00 on dist-test-slave-706q"
server_key: "023fce16c9573930853adae1b1a21585"
server_key_iv: "4ece6191943c70c5e27053447ac5182f"
server_key_version: "encryptionkey@0"
I20260620 09:30:00.366954 1165 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 09:30:00.367763 1180 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:30:00.367959 1165 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 09:30:00.368043 1165 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781947562949448-16558-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781947562949448-16558-0/minicluster-data/ts-0/wal
uuid: "5537713527764d9590472593f39e4f02"
format_stamp: "Formatted at 2026-06-20 09:30:00 on dist-test-slave-706q"
server_key: "023fce16c9573930853adae1b1a21585"
server_key_iv: "4ece6191943c70c5e27053447ac5182f"
server_key_version: "encryptionkey@0"
I20260620 09:30:00.368099 1165 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781947562949448-16558-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781947562949448-16558-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781947562949448-16558-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:30:00.387926 1165 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:30:00.389925 1165 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:30:00.390043 1165 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:30:00.390244 1165 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:30:00.390566 1165 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:30:00.390601 1165 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:30:00.390626 1165 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:30:00.390672 1165 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:30:00.398768 1165 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.129:40055
I20260620 09:30:00.398844 1293 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.129:40055 every 8 connection(s)
I20260620 09:30:00.399247 1165 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781947562949448-16558-0/minicluster-data/ts-0/data/info.pb
I20260620 09:30:00.402094 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 1165
I20260620 09:30:00.402236 16558 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781947562949448-16558-0/minicluster-data/ts-0/wal/instance
I20260620 09:30:00.402456 16558 external_mini_cluster.cc:1428] Setting key 2815e43ce37d131aaf10f0cb9b883faf
Jun 20 09:30:00 dist-test-slave-706q krb5kdc[1058](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947800, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.129@KRBTEST.COM for kudu/127.16.43.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.43.130@KRBTEST.COM; defaulting to no policy
I20260620 09:30:00.410917 1143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:00.400359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.129:33635 (local address 127.16.43.190:44157)
0620 09:30:00.400416 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:00.400418 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:00.400429 (+ 11us) server_negotiation.cc:407] Connection header received
0620 09:30:00.401106 (+ 677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:00.401107 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:00.401124 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:00.401183 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:30:00.402505 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:00.403249 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:00.404435 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:00.404667 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:00.404758 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:00.407376 (+ 2618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:30:00.407378 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:30:00.407378 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:30:00.407389 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:30:00.409423 (+ 2034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:30:00.409901 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:30:00.409902 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:30:00.409903 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:30:00.409970 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:30:00.410237 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:30:00.410238 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:30:00.410239 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:30:00.410392 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:30:00.410458 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:00.410764 (+ 306us) server_negotiation.cc:299] Negotiation successful
0620 09:30:00.410843 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 09:30:00.411123 1296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:00.400048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:44157 (local address 127.16.43.129:33635)
0620 09:30:00.400257 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0620 09:30:00.400311 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 09:30:00.400993 (+ 682us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:30:00.401267 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:30:00.401272 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:30:00.402095 (+ 823us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:30:00.402384 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:30:00.402391 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:30:00.403375 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:30:00.403376 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:30:00.404325 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:30:00.404327 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:30:00.404771 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:30:00.404772 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:30:00.404931 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:00.405534 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:30:00.405543 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:30:00.407244 (+ 1701us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:30:00.409533 (+ 2289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:30:00.409544 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:30:00.409546 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:30:00.409812 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:30:00.410040 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:30:00.410041 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:30:00.410042 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:30:00.410165 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:30:00.410493 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:30:00.410500 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:30:00.410641 (+ 141us) client_negotiation.cc:769] Sending connection context
0620 09:30:00.410732 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 09:30:00.410866 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":100,"threads_started":1}
I20260620 09:30:00.411824 1294 heartbeater.cc:347] Connected to a master server at 127.16.43.190:44157
I20260620 09:30:00.411921 1294 heartbeater.cc:464] Registering TS with master...
I20260620 09:30:00.412086 1294 heartbeater.cc:511] Master 127.16.43.190:44157 requested a full tablet report, sending...
I20260620 09:30:00.412640 1103 ts_manager.cc:194] Registered new tserver with Master: 5537713527764d9590472593f39e4f02 (127.16.43.129:40055)
I20260620 09:30:00.413339 1103 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.43.129@KRBTEST.COM'} at 127.16.43.129:33635
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.43.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.43.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.130.keytab.
Entry for principal kudu/127.16.43.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.130.keytab.
I20260620 09:30:00.431455 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781947562949448-16558-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781947562949448-16558-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781947562949448-16558-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781947562949448-16558-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.43.130:0
--local_ip_for_outbound_sockets=127.16.43.130
--webserver_interface=127.16.43.130
--webserver_port=0
--tserver_master_addrs=127.16.43.190:44157
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.130.keytab
--principal=kudu/127.16.43.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.43.148:35629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781947562949448-16558-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781947562949448-16558-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781947562949448-16558-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:30:00.514698 1299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:30:00.514928 1299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:30:00.514955 1299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:30:00.516222 1299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:30:00.516269 1299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:30:00.516297 1299 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.43.130
I20260620 09:30:00.517820 1299 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:35629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781947562949448-16558-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781947562949448-16558-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.130.keytab
--principal=kudu/127.16.43.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.43.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781947562949448-16558-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.43.130
--webserver_port=0
--tserver_master_addrs=127.16.43.190:44157
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1299
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.43.130
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781947562949448-16558-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:30:00.518078 1299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:30:00.518348 1299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:30:00.523728 1305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:30:00.523722 1307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:30:00.523722 1304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:30:00.524055 1299 server_base.cc:1034] running on GCE node
I20260620 09:30:00.524293 1299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:30:00.524507 1299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:30:00.525730 1299 hybrid_clock.cc:648] HybridClock initialized: now 1781947800525712 us; error 30 us; skew 500 ppm
Jun 20 09:30:00 dist-test-slave-706q krb5kdc[1058](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947800, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:30:00.528649 1299 init.cc:377] Logged in from keytab as kudu/127.16.43.130@KRBTEST.COM (short username kudu)
I20260620 09:30:00.529161 1299 webserver.cc:460] Webserver started at http://127.16.43.130:40311/ using document root <none> and password file <none>
I20260620 09:30:00.529366 1299 fs_manager.cc:359] Metadata directory not provided
I20260620 09:30:00.529409 1299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:30:00.529510 1299 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:30:00.530638 1299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781947562949448-16558-0/minicluster-data/ts-1/data/instance:
uuid: "8c547b31ddca4bbd891d42f3ca09e0a1"
format_stamp: "Formatted at 2026-06-20 09:30:00 on dist-test-slave-706q"
server_key: "04ca865ef54d508d502a05baa5b6a133"
server_key_iv: "b745d213bdc130ec09ce302fa69495be"
server_key_version: "encryptionkey@0"
I20260620 09:30:00.530965 1299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781947562949448-16558-0/minicluster-data/ts-1/wal/instance:
uuid: "8c547b31ddca4bbd891d42f3ca09e0a1"
format_stamp: "Formatted at 2026-06-20 09:30:00 on dist-test-slave-706q"
server_key: "04ca865ef54d508d502a05baa5b6a133"
server_key_iv: "b745d213bdc130ec09ce302fa69495be"
server_key_version: "encryptionkey@0"
I20260620 09:30:00.532512 1299 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.003s
I20260620 09:30:00.533358 1314 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:30:00.533558 1299 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:30:00.533689 1299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781947562949448-16558-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781947562949448-16558-0/minicluster-data/ts-1/wal
uuid: "8c547b31ddca4bbd891d42f3ca09e0a1"
format_stamp: "Formatted at 2026-06-20 09:30:00 on dist-test-slave-706q"
server_key: "04ca865ef54d508d502a05baa5b6a133"
server_key_iv: "b745d213bdc130ec09ce302fa69495be"
server_key_version: "encryptionkey@0"
I20260620 09:30:00.533749 1299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781947562949448-16558-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781947562949448-16558-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781947562949448-16558-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:30:00.543821 1299 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:30:00.546139 1299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:30:00.546255 1299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:30:00.546442 1299 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:30:00.546767 1299 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:30:00.546803 1299 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:30:00.546828 1299 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:30:00.546840 1299 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:30:00.552850 1299 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.130:35841
I20260620 09:30:00.552924 1427 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.130:35841 every 8 connection(s)
I20260620 09:30:00.553231 1299 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781947562949448-16558-0/minicluster-data/ts-1/data/info.pb
I20260620 09:30:00.556824 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 1299
I20260620 09:30:00.556941 16558 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781947562949448-16558-0/minicluster-data/ts-1/wal/instance
I20260620 09:30:00.557185 16558 external_mini_cluster.cc:1428] Setting key 2ee0ac74df677aa77a002f908f9c8b19
Jun 20 09:30:00 dist-test-slave-706q krb5kdc[1058](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947800, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.130@KRBTEST.COM for kudu/127.16.43.190@KRBTEST.COM
I20260620 09:30:00.563797 1143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:00.553980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.130:57511 (local address 127.16.43.190:44157)
0620 09:30:00.554050 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:00.554053 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:00.554294 (+ 241us) server_negotiation.cc:407] Connection header received
0620 09:30:00.554875 (+ 581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:00.554876 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:00.554886 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:00.554936 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:30:00.555979 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:00.556473 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:00.557907 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:00.558162 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:00.558257 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:00.560736 (+ 2479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:30:00.560738 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:30:00.560738 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:30:00.560747 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:30:00.562296 (+ 1549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:30:00.562733 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:30:00.562734 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:30:00.562734 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:30:00.562796 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:30:00.563055 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:30:00.563056 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:30:00.563057 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:30:00.563207 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:30:00.563272 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:00.563609 (+ 337us) server_negotiation.cc:299] Negotiation successful
0620 09:30:00.563701 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 09:30:00.564038 1430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:00.553966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:44157 (local address 127.16.43.130:57511)
0620 09:30:00.554174 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0620 09:30:00.554209 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 09:30:00.554736 (+ 527us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:30:00.555013 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:30:00.555016 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:30:00.555663 (+ 647us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:30:00.555887 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:30:00.555893 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:30:00.556557 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:30:00.556558 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:30:00.557807 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:30:00.557809 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:30:00.558266 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:30:00.558267 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:30:00.558393 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:00.558940 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:30:00.558948 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:30:00.560627 (+ 1679us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:30:00.562387 (+ 1760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:30:00.562393 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:30:00.562395 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:30:00.562633 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:30:00.562865 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:30:00.562866 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:30:00.562866 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:30:00.562987 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:30:00.563263 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:30:00.563289 (+ 26us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:30:00.563435 (+ 146us) client_negotiation.cc:769] Sending connection context
0620 09:30:00.563585 (+ 150us) client_negotiation.cc:240] Negotiation successful
0620 09:30:00.563762 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":118,"threads_started":1}
I20260620 09:30:00.564507 1428 heartbeater.cc:347] Connected to a master server at 127.16.43.190:44157
I20260620 09:30:00.564621 1428 heartbeater.cc:464] Registering TS with master...
I20260620 09:30:00.564806 1428 heartbeater.cc:511] Master 127.16.43.190:44157 requested a full tablet report, sending...
I20260620 09:30:00.565233 1103 ts_manager.cc:194] Registered new tserver with Master: 8c547b31ddca4bbd891d42f3ca09e0a1 (127.16.43.130:35841)
WARNING: no policy specified for kudu/127.16.43.131@KRBTEST.COM; defaulting to no policy
I20260620 09:30:00.565673 1103 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.43.130@KRBTEST.COM'} at 127.16.43.130:57511
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.43.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.43.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.131.keytab.
Entry for principal kudu/127.16.43.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.131.keytab.
I20260620 09:30:00.584075 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781947562949448-16558-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781947562949448-16558-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781947562949448-16558-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781947562949448-16558-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.43.131:0
--local_ip_for_outbound_sockets=127.16.43.131
--webserver_interface=127.16.43.131
--webserver_port=0
--tserver_master_addrs=127.16.43.190:44157
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.131.keytab
--principal=kudu/127.16.43.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.43.148:35629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781947562949448-16558-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781947562949448-16558-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781947562949448-16558-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:30:00.668238 1433 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:30:00.668457 1433 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:30:00.668484 1433 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:30:00.670179 1433 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:30:00.670238 1433 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:30:00.670269 1433 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.43.131
I20260620 09:30:00.672017 1433 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:35629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781947562949448-16558-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781947562949448-16558-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.131.keytab
--principal=kudu/127.16.43.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.43.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781947562949448-16558-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.43.131
--webserver_port=0
--tserver_master_addrs=127.16.43.190:44157
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1433
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.43.131
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781947562949448-16558-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:30:00.672228 1433 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:30:00.672416 1433 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 09:30:00.677934 1433 server_base.cc:1034] running on GCE node
W20260620 09:30:00.677891 1441 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:30:00.677876 1439 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:30:00.677871 1438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:30:00.678378 1433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:30:00.678726 1433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:30:00.679919 1433 hybrid_clock.cc:648] HybridClock initialized: now 1781947800679905 us; error 48 us; skew 500 ppm
Jun 20 09:30:00 dist-test-slave-706q krb5kdc[1058](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947800, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:30:00.682583 1433 init.cc:377] Logged in from keytab as kudu/127.16.43.131@KRBTEST.COM (short username kudu)
I20260620 09:30:00.683065 1433 webserver.cc:460] Webserver started at http://127.16.43.131:45327/ using document root <none> and password file <none>
I20260620 09:30:00.683311 1433 fs_manager.cc:359] Metadata directory not provided
I20260620 09:30:00.683358 1433 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:30:00.683516 1433 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:30:00.684650 1433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781947562949448-16558-0/minicluster-data/ts-2/data/instance:
uuid: "23b67de5d91a47a782d84d24fef4b65e"
format_stamp: "Formatted at 2026-06-20 09:30:00 on dist-test-slave-706q"
server_key: "3a70ad00418c2a956fa45c4858bd4010"
server_key_iv: "037b328e9965d6ff8eccec0ea6f062e7"
server_key_version: "encryptionkey@0"
I20260620 09:30:00.685083 1433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781947562949448-16558-0/minicluster-data/ts-2/wal/instance:
uuid: "23b67de5d91a47a782d84d24fef4b65e"
format_stamp: "Formatted at 2026-06-20 09:30:00 on dist-test-slave-706q"
server_key: "3a70ad00418c2a956fa45c4858bd4010"
server_key_iv: "037b328e9965d6ff8eccec0ea6f062e7"
server_key_version: "encryptionkey@0"
I20260620 09:30:00.686970 1433 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260620 09:30:00.687811 1448 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:30:00.688026 1433 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 09:30:00.688097 1433 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781947562949448-16558-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781947562949448-16558-0/minicluster-data/ts-2/wal
uuid: "23b67de5d91a47a782d84d24fef4b65e"
format_stamp: "Formatted at 2026-06-20 09:30:00 on dist-test-slave-706q"
server_key: "3a70ad00418c2a956fa45c4858bd4010"
server_key_iv: "037b328e9965d6ff8eccec0ea6f062e7"
server_key_version: "encryptionkey@0"
I20260620 09:30:00.688192 1433 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781947562949448-16558-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781947562949448-16558-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781947562949448-16558-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:30:00.707259 1433 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:30:00.709710 1433 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:30:00.709854 1433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:30:00.710117 1433 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:30:00.710435 1433 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:30:00.710489 1433 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:30:00.710532 1433 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:30:00.710558 1433 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:30:00.716811 1433 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.131:41995
I20260620 09:30:00.716856 1561 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.131:41995 every 8 connection(s)
I20260620 09:30:00.717192 1433 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781947562949448-16558-0/minicluster-data/ts-2/data/info.pb
I20260620 09:30:00.719810 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 1433
I20260620 09:30:00.719918 16558 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781947562949448-16558-0/minicluster-data/ts-2/wal/instance
I20260620 09:30:00.720103 16558 external_mini_cluster.cc:1428] Setting key 105a872a6ba600bf458e766272976a3a
Jun 20 09:30:00 dist-test-slave-706q krb5kdc[1058](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947800, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.131@KRBTEST.COM for kudu/127.16.43.190@KRBTEST.COM
I20260620 09:30:00.727916 1143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:00.717980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.131:36355 (local address 127.16.43.190:44157)
0620 09:30:00.718051 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:00.718053 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:00.718220 (+ 167us) server_negotiation.cc:407] Connection header received
0620 09:30:00.718789 (+ 569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:00.718790 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:00.718804 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:00.718868 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:30:00.719970 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:00.720733 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:00.721985 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:00.722186 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:00.722249 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:00.724696 (+ 2447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:30:00.724698 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:30:00.724698 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:30:00.724706 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:30:00.726241 (+ 1535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:30:00.726738 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:30:00.726739 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:30:00.726740 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:30:00.726798 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:30:00.727117 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:30:00.727119 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:30:00.727119 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:30:00.727256 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:30:00.727322 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:00.727758 (+ 436us) server_negotiation.cc:299] Negotiation successful
0620 09:30:00.727838 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 09:30:00.728098 1564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:00.717918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:44157 (local address 127.16.43.131:36355)
0620 09:30:00.718089 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0620 09:30:00.718132 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 09:30:00.718672 (+ 540us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:30:00.718941 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:30:00.718946 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:30:00.719576 (+ 630us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:30:00.719809 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:30:00.719814 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:30:00.720854 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:30:00.720855 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:30:00.721891 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:30:00.721894 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:30:00.722272 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:30:00.722272 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:30:00.722383 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:00.722931 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:30:00.722940 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:30:00.724575 (+ 1635us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:30:00.726344 (+ 1769us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:30:00.726351 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:30:00.726354 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:30:00.726639 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:30:00.726889 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:30:00.726890 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:30:00.726890 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:30:00.727015 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:30:00.727331 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:30:00.727337 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:30:00.727472 (+ 135us) client_negotiation.cc:769] Sending connection context
0620 09:30:00.727697 (+ 225us) client_negotiation.cc:240] Negotiation successful
0620 09:30:00.727853 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":103,"threads_started":1}
I20260620 09:30:00.728719 1562 heartbeater.cc:347] Connected to a master server at 127.16.43.190:44157
I20260620 09:30:00.728834 1562 heartbeater.cc:464] Registering TS with master...
I20260620 09:30:00.729039 1562 heartbeater.cc:511] Master 127.16.43.190:44157 requested a full tablet report, sending...
I20260620 09:30:00.729504 1103 ts_manager.cc:194] Registered new tserver with Master: 23b67de5d91a47a782d84d24fef4b65e (127.16.43.131:41995)
I20260620 09:30:00.729868 1103 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.43.131@KRBTEST.COM'} at 127.16.43.131:36355
I20260620 09:30:00.732321 16558 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 09:30:00.740268 1143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:00.733742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38524 (local address 127.16.43.190:44157)
0620 09:30:00.733832 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:00.733834 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:00.733885 (+ 51us) server_negotiation.cc:407] Connection header received
0620 09:30:00.733949 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:00.733949 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:00.733959 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:00.734010 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:30:00.734554 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:00.735047 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:00.735954 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:00.736180 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:00.736249 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:00.737514 (+ 1265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:30:00.737516 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:30:00.737516 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:30:00.737522 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:30:00.738984 (+ 1462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:30:00.739319 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:30:00.739320 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:30:00.739321 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:30:00.739371 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:30:00.739581 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:30:00.739582 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:30:00.739582 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:30:00.739756 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:30:00.739834 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:00.740022 (+ 188us) server_negotiation.cc:299] Negotiation successful
0620 09:30:00.740152 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 09:30:00.742054 1103 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38524:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 09:30:00.743626 1103 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 09:30:00.750753 1577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:00.746247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44572 (local address 127.16.43.129:40055)
0620 09:30:00.746442 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:00.746447 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:00.746456 (+ 9us) server_negotiation.cc:407] Connection header received
0620 09:30:00.746501 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:00.746502 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:00.746538 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:00.746621 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:30:00.746840 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:00.748046 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:00.749614 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:00.750360 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:00.750440 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:00.750499 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:00.750650 (+ 151us) server_negotiation.cc:299] Negotiation successful
0620 09:30:00.750686 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":84,"threads_started":1}
I20260620 09:30:00.750764 1574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:00.746034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.129:40055 (local address 127.0.0.1:44572)
0620 09:30:00.746220 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0620 09:30:00.746262 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 09:30:00.746332 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:30:00.746653 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:30:00.746656 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:30:00.746658 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:30:00.746748 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:30:00.746750 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:30:00.748132 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:30:00.748132 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:30:00.749510 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:30:00.749512 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:30:00.750446 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:30:00.750446 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:30:00.750525 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:00.750541 (+ 16us) client_negotiation.cc:769] Sending connection context
0620 09:30:00.750617 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 09:30:00.750672 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"mutex_wait_us":49,"thread_start_us":89,"threads_started":1}
I20260620 09:30:00.751112 1575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:00.746034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.130:35841 (local address 127.0.0.1:51348)
0620 09:30:00.746481 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0620 09:30:00.746491 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 09:30:00.746639 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:30:00.747029 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:30:00.747030 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:30:00.747032 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:30:00.747077 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:30:00.747079 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:30:00.748355 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:30:00.748356 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:30:00.749509 (+ 1153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:30:00.749512 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:30:00.750886 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:30:00.750887 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:30:00.750951 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:00.750956 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 09:30:00.751006 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 09:30:00.751044 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":425,"mutex_wait_us":13,"thread_start_us":59,"threads_started":1}
I20260620 09:30:00.751366 1576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:00.746240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51348 (local address 127.16.43.130:35841)
0620 09:30:00.746416 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:00.746432 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:00.746701 (+ 269us) server_negotiation.cc:407] Connection header received
0620 09:30:00.746744 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:00.746747 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:00.746763 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:00.746951 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:30:00.747154 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:00.748269 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:00.749614 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:00.750360 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:00.750440 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:00.750499 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:00.751231 (+ 732us) server_negotiation.cc:299] Negotiation successful
0620 09:30:00.751280 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":72,"threads_started":1}
I20260620 09:30:00.751614 1573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:00.746070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.131:41995 (local address 127.0.0.1:47868)
0620 09:30:00.746717 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0620 09:30:00.746726 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 09:30:00.746806 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:30:00.747066 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:30:00.747067 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:30:00.747069 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:30:00.747117 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:30:00.747119 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:30:00.748045 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:30:00.748046 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:30:00.750215 (+ 2169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:30:00.750218 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:30:00.751335 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:30:00.751336 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:30:00.751409 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:00.751414 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 09:30:00.751463 (+ 49us) client_negotiation.cc:240] Negotiation successful
0620 09:30:00.751504 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":635,"mutex_wait_us":1322,"thread_start_us":110,"threads_started":1}
I20260620 09:30:00.751591 1228 tablet_service.cc:1467] Processing CreateTablet for tablet c9e60efa19a5402e82bf6a9ccc28efa5 (DEFAULT_TABLE table=test-table [id=a4614f011b894b8899840ee4a71bd0c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:30:00.751924 1578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:00.746424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47868 (local address 127.16.43.131:41995)
0620 09:30:00.746613 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:00.746616 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:00.746798 (+ 182us) server_negotiation.cc:407] Connection header received
0620 09:30:00.746879 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:00.746882 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:00.746895 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:00.746987 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:30:00.747200 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:00.747944 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:00.750312 (+ 2368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:00.751183 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:00.751248 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:00.751303 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:00.751796 (+ 493us) server_negotiation.cc:299] Negotiation successful
0620 09:30:00.751841 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":89,"threads_started":1}
I20260620 09:30:00.751986 1228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9e60efa19a5402e82bf6a9ccc28efa5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:30:00.752096 1362 tablet_service.cc:1467] Processing CreateTablet for tablet c9e60efa19a5402e82bf6a9ccc28efa5 (DEFAULT_TABLE table=test-table [id=a4614f011b894b8899840ee4a71bd0c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:30:00.752434 1362 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9e60efa19a5402e82bf6a9ccc28efa5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:30:00.752480 1496 tablet_service.cc:1467] Processing CreateTablet for tablet c9e60efa19a5402e82bf6a9ccc28efa5 (DEFAULT_TABLE table=test-table [id=a4614f011b894b8899840ee4a71bd0c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:30:00.752782 1496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9e60efa19a5402e82bf6a9ccc28efa5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:30:00.755038 1579 tablet_bootstrap.cc:492] T c9e60efa19a5402e82bf6a9ccc28efa5 P 5537713527764d9590472593f39e4f02: Bootstrap starting.
I20260620 09:30:00.755729 1579 tablet_bootstrap.cc:654] T c9e60efa19a5402e82bf6a9ccc28efa5 P 5537713527764d9590472593f39e4f02: Neither blocks nor log segments found. Creating new log.
I20260620 09:30:00.756042 1580 tablet_bootstrap.cc:492] T c9e60efa19a5402e82bf6a9ccc28efa5 P 8c547b31ddca4bbd891d42f3ca09e0a1: Bootstrap starting.
I20260620 09:30:00.756110 1581 tablet_bootstrap.cc:492] T c9e60efa19a5402e82bf6a9ccc28efa5 P 23b67de5d91a47a782d84d24fef4b65e: Bootstrap starting.
I20260620 09:30:00.756325 1579 log.cc:824] T c9e60efa19a5402e82bf6a9ccc28efa5 P 5537713527764d9590472593f39e4f02: Log is configured to *not* fsync() on all Append() calls
I20260620 09:30:00.756740 1580 tablet_bootstrap.cc:654] T c9e60efa19a5402e82bf6a9ccc28efa5 P 8c547b31ddca4bbd891d42f3ca09e0a1: Neither blocks nor log segments found. Creating new log.
I20260620 09:30:00.756742 1581 tablet_bootstrap.cc:654] T c9e60efa19a5402e82bf6a9ccc28efa5 P 23b67de5d91a47a782d84d24fef4b65e: Neither blocks nor log segments found. Creating new log.
I20260620 09:30:00.757057 1579 tablet_bootstrap.cc:492] T c9e60efa19a5402e82bf6a9ccc28efa5 P 5537713527764d9590472593f39e4f02: No bootstrap required, opened a new log
I20260620 09:30:00.757139 1579 ts_tablet_manager.cc:1397] T c9e60efa19a5402e82bf6a9ccc28efa5 P 5537713527764d9590472593f39e4f02: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 09:30:00.757323 1581 log.cc:824] T c9e60efa19a5402e82bf6a9ccc28efa5 P 23b67de5d91a47a782d84d24fef4b65e: Log is configured to *not* fsync() on all Append() calls
I20260620 09:30:00.757360 1580 log.cc:824] T c9e60efa19a5402e82bf6a9ccc28efa5 P 8c547b31ddca4bbd891d42f3ca09e0a1: Log is configured to *not* fsync() on all Append() calls
I20260620 09:30:00.758057 1581 tablet_bootstrap.cc:492] T c9e60efa19a5402e82bf6a9ccc28efa5 P 23b67de5d91a47a782d84d24fef4b65e: No bootstrap required, opened a new log
I20260620 09:30:00.758132 1581 ts_tablet_manager.cc:1397] T c9e60efa19a5402e82bf6a9ccc28efa5 P 23b67de5d91a47a782d84d24fef4b65e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 09:30:00.758273 1580 tablet_bootstrap.cc:492] T c9e60efa19a5402e82bf6a9ccc28efa5 P 8c547b31ddca4bbd891d42f3ca09e0a1: No bootstrap required, opened a new log
I20260620 09:30:00.758329 1580 ts_tablet_manager.cc:1397] T c9e60efa19a5402e82bf6a9ccc28efa5 P 8c547b31ddca4bbd891d42f3ca09e0a1: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 09:30:00.759083 1579 raft_consensus.cc:348] T c9e60efa19a5402e82bf6a9ccc28efa5 P 5537713527764d9590472593f39e4f02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5537713527764d9590472593f39e4f02" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 40055 } } peers { permanent_uuid: "8c547b31ddca4bbd891d42f3ca09e0a1" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 35841 } } peers { permanent_uuid: "23b67de5d91a47a782d84d24fef4b65e" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 41995 } }
I20260620 09:30:00.759248 1579 raft_consensus.cc:374] T c9e60efa19a5402e82bf6a9ccc28efa5 P 5537713527764d9590472593f39e4f02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:30:00.759290 1579 raft_consensus.cc:729] T c9e60efa19a5402e82bf6a9ccc28efa5 P 5537713527764d9590472593f39e4f02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5537713527764d9590472593f39e4f02, State: Initialized, Role: FOLLOWER
I20260620 09:30:00.759430 1579 consensus_queue.cc:260] T c9e60efa19a5402e82bf6a9ccc28efa5 P 5537713527764d9590472593f39e4f02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5537713527764d9590472593f39e4f02" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 40055 } } peers { permanent_uuid: "8c547b31ddca4bbd891d42f3ca09e0a1" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 35841 } } peers { permanent_uuid: "23b67de5d91a47a782d84d24fef4b65e" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 41995 } }
I20260620 09:30:00.759725 1579 ts_tablet_manager.cc:1428] T c9e60efa19a5402e82bf6a9ccc28efa5 P 5537713527764d9590472593f39e4f02: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 09:30:00.759887 1294 heartbeater.cc:503] Master 127.16.43.190:44157 was elected leader, sending a full tablet report...
I20260620 09:30:00.760182 1581 raft_consensus.cc:348] T c9e60efa19a5402e82bf6a9ccc28efa5 P 23b67de5d91a47a782d84d24fef4b65e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5537713527764d9590472593f39e4f02" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 40055 } } peers { permanent_uuid: "8c547b31ddca4bbd891d42f3ca09e0a1" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 35841 } } peers { permanent_uuid: "23b67de5d91a47a782d84d24fef4b65e" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 41995 } }
I20260620 09:30:00.760244 1580 raft_consensus.cc:348] T c9e60efa19a5402e82bf6a9ccc28efa5 P 8c547b31ddca4bbd891d42f3ca09e0a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5537713527764d9590472593f39e4f02" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 40055 } } peers { permanent_uuid: "8c547b31ddca4bbd891d42f3ca09e0a1" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 35841 } } peers { permanent_uuid: "23b67de5d91a47a782d84d24fef4b65e" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 41995 } }
I20260620 09:30:00.760358 1581 raft_consensus.cc:374] T c9e60efa19a5402e82bf6a9ccc28efa5 P 23b67de5d91a47a782d84d24fef4b65e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:30:00.760386 1580 raft_consensus.cc:374] T c9e60efa19a5402e82bf6a9ccc28efa5 P 8c547b31ddca4bbd891d42f3ca09e0a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:30:00.760397 1581 raft_consensus.cc:729] T c9e60efa19a5402e82bf6a9ccc28efa5 P 23b67de5d91a47a782d84d24fef4b65e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23b67de5d91a47a782d84d24fef4b65e, State: Initialized, Role: FOLLOWER
I20260620 09:30:00.760430 1580 raft_consensus.cc:729] T c9e60efa19a5402e82bf6a9ccc28efa5 P 8c547b31ddca4bbd891d42f3ca09e0a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c547b31ddca4bbd891d42f3ca09e0a1, State: Initialized, Role: FOLLOWER
I20260620 09:30:00.760532 1581 consensus_queue.cc:260] T c9e60efa19a5402e82bf6a9ccc28efa5 P 23b67de5d91a47a782d84d24fef4b65e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5537713527764d9590472593f39e4f02" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 40055 } } peers { permanent_uuid: "8c547b31ddca4bbd891d42f3ca09e0a1" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 35841 } } peers { permanent_uuid: "23b67de5d91a47a782d84d24fef4b65e" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 41995 } }
I20260620 09:30:00.760531 1580 consensus_queue.cc:260] T c9e60efa19a5402e82bf6a9ccc28efa5 P 8c547b31ddca4bbd891d42f3ca09e0a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5537713527764d9590472593f39e4f02" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 40055 } } peers { permanent_uuid: "8c547b31ddca4bbd891d42f3ca09e0a1" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 35841 } } peers { permanent_uuid: "23b67de5d91a47a782d84d24fef4b65e" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 41995 } }
I20260620 09:30:00.760797 1580 ts_tablet_manager.cc:1428] T c9e60efa19a5402e82bf6a9ccc28efa5 P 8c547b31ddca4bbd891d42f3ca09e0a1: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 09:30:00.760797 1581 ts_tablet_manager.cc:1428] T c9e60efa19a5402e82bf6a9ccc28efa5 P 23b67de5d91a47a782d84d24fef4b65e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 09:30:00.760907 1562 heartbeater.cc:503] Master 127.16.43.190:44157 was elected leader, sending a full tablet report...
I20260620 09:30:00.760912 1428 heartbeater.cc:503] Master 127.16.43.190:44157 was elected leader, sending a full tablet report...
W20260620 09:30:00.803790 1429 tablet.cc:2369] T c9e60efa19a5402e82bf6a9ccc28efa5 P 8c547b31ddca4bbd891d42f3ca09e0a1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 09:30:00.900444 1295 tablet.cc:2369] T c9e60efa19a5402e82bf6a9ccc28efa5 P 5537713527764d9590472593f39e4f02: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 09:30:00.967882 1563 tablet.cc:2369] T c9e60efa19a5402e82bf6a9ccc28efa5 P 23b67de5d91a47a782d84d24fef4b65e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 09:30:01.008730 1586 raft_consensus.cc:482] T c9e60efa19a5402e82bf6a9ccc28efa5 P 23b67de5d91a47a782d84d24fef4b65e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:30:01.008863 1586 raft_consensus.cc:504] T c9e60efa19a5402e82bf6a9ccc28efa5 P 23b67de5d91a47a782d84d24fef4b65e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5537713527764d9590472593f39e4f02" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 40055 } } peers { permanent_uuid: "8c547b31ddca4bbd891d42f3ca09e0a1" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 35841 } } peers { permanent_uuid: "23b67de5d91a47a782d84d24fef4b65e" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 41995 } }
I20260620 09:30:01.009325 1586 leader_election.cc:288] T c9e60efa19a5402e82bf6a9ccc28efa5 P 23b67de5d91a47a782d84d24fef4b65e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5537713527764d9590472593f39e4f02 (127.16.43.129:40055), 8c547b31ddca4bbd891d42f3ca09e0a1 (127.16.43.130:35841)
I20260620 09:30:01.013445 1576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:01.009710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.131:48487 (local address 127.16.43.130:35841)
0620 09:30:01.009789 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:01.009791 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:01.009802 (+ 11us) server_negotiation.cc:407] Connection header received
0620 09:30:01.009841 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:01.009843 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:01.009856 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:01.009935 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:30:01.010127 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:01.010909 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:01.012464 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:01.013039 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:01.013113 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:01.013122 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:01.013327 (+ 205us) server_negotiation.cc:299] Negotiation successful
0620 09:30:01.013370 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 09:30:01.013445 1577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:01.009651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.131:39071 (local address 127.16.43.129:40055)
0620 09:30:01.009700 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:01.009702 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:01.010057 (+ 355us) server_negotiation.cc:407] Connection header received
0620 09:30:01.010120 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:01.010120 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:01.010134 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:01.010192 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:30:01.010344 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:01.011012 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:01.012464 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:01.013024 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:01.013114 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:01.013123 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:01.013327 (+ 204us) server_negotiation.cc:299] Negotiation successful
0620 09:30:01.013372 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 09:30:01.013448 1564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:01.009568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.130:35841 (local address 127.16.43.131:48487)
0620 09:30:01.009636 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0620 09:30:01.009649 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 09:30:01.009703 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:30:01.009959 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:30:01.009960 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:30:01.009962 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:30:01.010038 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:30:01.010040 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:30:01.011016 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:30:01.011017 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:30:01.012339 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:30:01.012341 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:30:01.013129 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:30:01.013130 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:30:01.013230 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:01.013233 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 09:30:01.013285 (+ 52us) client_negotiation.cc:240] Negotiation successful
0620 09:30:01.013346 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"mutex_wait_us":34}
I20260620 09:30:01.013465 1588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:01.009582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.129:40055 (local address 127.16.43.131:39071)
0620 09:30:01.009965 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0620 09:30:01.009976 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 09:30:01.010034 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:30:01.010220 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:30:01.010220 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:30:01.010222 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:30:01.010272 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:30:01.010273 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:30:01.011092 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:30:01.011093 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:30:01.012338 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:30:01.012341 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:30:01.013133 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:30:01.013133 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:30:01.013220 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:01.013224 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 09:30:01.013283 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 09:30:01.013346 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":366,"mutex_wait_us":202,"thread_start_us":105,"threads_started":1}
I20260620 09:30:01.013866 1382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9e60efa19a5402e82bf6a9ccc28efa5" candidate_uuid: "23b67de5d91a47a782d84d24fef4b65e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c547b31ddca4bbd891d42f3ca09e0a1" is_pre_election: true
I20260620 09:30:01.014050 1382 raft_consensus.cc:2454] T c9e60efa19a5402e82bf6a9ccc28efa5 P 8c547b31ddca4bbd891d42f3ca09e0a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23b67de5d91a47a782d84d24fef4b65e in term 0.
I20260620 09:30:01.014056 1248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9e60efa19a5402e82bf6a9ccc28efa5" candidate_uuid: "23b67de5d91a47a782d84d24fef4b65e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5537713527764d9590472593f39e4f02" is_pre_election: true
I20260620 09:30:01.014197 1248 raft_consensus.cc:2454] T c9e60efa19a5402e82bf6a9ccc28efa5 P 5537713527764d9590472593f39e4f02 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23b67de5d91a47a782d84d24fef4b65e in term 0.
I20260620 09:30:01.014433 1449 leader_election.cc:302] T c9e60efa19a5402e82bf6a9ccc28efa5 P 23b67de5d91a47a782d84d24fef4b65e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23b67de5d91a47a782d84d24fef4b65e, 5537713527764d9590472593f39e4f02; no voters:
I20260620 09:30:01.014559 1586 raft_consensus.cc:2781] T c9e60efa19a5402e82bf6a9ccc28efa5 P 23b67de5d91a47a782d84d24fef4b65e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 09:30:01.014613 1586 raft_consensus.cc:482] T c9e60efa19a5402e82bf6a9ccc28efa5 P 23b67de5d91a47a782d84d24fef4b65e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 09:30:01.014639 1586 raft_consensus.cc:3037] T c9e60efa19a5402e82bf6a9ccc28efa5 P 23b67de5d91a47a782d84d24fef4b65e [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:30:01.015367 1586 raft_consensus.cc:504] T c9e60efa19a5402e82bf6a9ccc28efa5 P 23b67de5d91a47a782d84d24fef4b65e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5537713527764d9590472593f39e4f02" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 40055 } } peers { permanent_uuid: "8c547b31ddca4bbd891d42f3ca09e0a1" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 35841 } } peers { permanent_uuid: "23b67de5d91a47a782d84d24fef4b65e" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 41995 } }
I20260620 09:30:01.015514 1586 leader_election.cc:288] T c9e60efa19a5402e82bf6a9ccc28efa5 P 23b67de5d91a47a782d84d24fef4b65e [CANDIDATE]: Term 1 election: Requested vote from peers 5537713527764d9590472593f39e4f02 (127.16.43.129:40055), 8c547b31ddca4bbd891d42f3ca09e0a1 (127.16.43.130:35841)
I20260620 09:30:01.015704 1248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9e60efa19a5402e82bf6a9ccc28efa5" candidate_uuid: "23b67de5d91a47a782d84d24fef4b65e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5537713527764d9590472593f39e4f02"
I20260620 09:30:01.015720 1382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9e60efa19a5402e82bf6a9ccc28efa5" candidate_uuid: "23b67de5d91a47a782d84d24fef4b65e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c547b31ddca4bbd891d42f3ca09e0a1"
I20260620 09:30:01.015810 1248 raft_consensus.cc:3037] T c9e60efa19a5402e82bf6a9ccc28efa5 P 5537713527764d9590472593f39e4f02 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:30:01.015805 1382 raft_consensus.cc:3037] T c9e60efa19a5402e82bf6a9ccc28efa5 P 8c547b31ddca4bbd891d42f3ca09e0a1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:30:01.016510 1382 raft_consensus.cc:2454] T c9e60efa19a5402e82bf6a9ccc28efa5 P 8c547b31ddca4bbd891d42f3ca09e0a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23b67de5d91a47a782d84d24fef4b65e in term 1.
I20260620 09:30:01.016535 1248 raft_consensus.cc:2454] T c9e60efa19a5402e82bf6a9ccc28efa5 P 5537713527764d9590472593f39e4f02 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23b67de5d91a47a782d84d24fef4b65e in term 1.
I20260620 09:30:01.016696 1449 leader_election.cc:302] T c9e60efa19a5402e82bf6a9ccc28efa5 P 23b67de5d91a47a782d84d24fef4b65e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23b67de5d91a47a782d84d24fef4b65e, 5537713527764d9590472593f39e4f02; no voters:
I20260620 09:30:01.016855 1586 raft_consensus.cc:2781] T c9e60efa19a5402e82bf6a9ccc28efa5 P 23b67de5d91a47a782d84d24fef4b65e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:30:01.016999 1586 raft_consensus.cc:686] T c9e60efa19a5402e82bf6a9ccc28efa5 P 23b67de5d91a47a782d84d24fef4b65e [term 1 LEADER]: Becoming Leader. State: Replica: 23b67de5d91a47a782d84d24fef4b65e, State: Running, Role: LEADER
I20260620 09:30:01.017096 1586 consensus_queue.cc:237] T c9e60efa19a5402e82bf6a9ccc28efa5 P 23b67de5d91a47a782d84d24fef4b65e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5537713527764d9590472593f39e4f02" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 40055 } } peers { permanent_uuid: "8c547b31ddca4bbd891d42f3ca09e0a1" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 35841 } } peers { permanent_uuid: "23b67de5d91a47a782d84d24fef4b65e" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 41995 } }
I20260620 09:30:01.017928 1103 catalog_manager.cc:5495] T c9e60efa19a5402e82bf6a9ccc28efa5 P 23b67de5d91a47a782d84d24fef4b65e reported cstate change: term changed from 0 to 1, leader changed from <none> to 23b67de5d91a47a782d84d24fef4b65e (127.16.43.131). New cstate: current_term: 1 leader_uuid: "23b67de5d91a47a782d84d24fef4b65e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5537713527764d9590472593f39e4f02" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 40055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c547b31ddca4bbd891d42f3ca09e0a1" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 35841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "23b67de5d91a47a782d84d24fef4b65e" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 41995 } health_report { overall_health: HEALTHY } } }
I20260620 09:30:01.033744 1578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:01.031124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47872 (local address 127.16.43.131:41995)
0620 09:30:01.031192 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:01.031194 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:01.031204 (+ 10us) server_negotiation.cc:407] Connection header received
0620 09:30:01.031276 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:01.031277 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:01.031293 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:01.031404 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:30:01.031579 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:01.032064 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:01.032887 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:01.033080 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:01.033168 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:01.033368 (+ 200us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:30:01.033479 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:30:01.033577 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:01.033602 (+ 25us) server_negotiation.cc:299] Negotiation successful
0620 09:30:01.033623 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 09:30:01.035945 1248 raft_consensus.cc:1261] T c9e60efa19a5402e82bf6a9ccc28efa5 P 5537713527764d9590472593f39e4f02 [term 1 FOLLOWER]: Refusing update from remote peer 23b67de5d91a47a782d84d24fef4b65e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:30:01.035945 1382 raft_consensus.cc:1261] T c9e60efa19a5402e82bf6a9ccc28efa5 P 8c547b31ddca4bbd891d42f3ca09e0a1 [term 1 FOLLOWER]: Refusing update from remote peer 23b67de5d91a47a782d84d24fef4b65e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:30:01.036360 1589 consensus_queue.cc:1035] T c9e60efa19a5402e82bf6a9ccc28efa5 P 23b67de5d91a47a782d84d24fef4b65e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5537713527764d9590472593f39e4f02" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 40055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:30:01.036482 1586 consensus_queue.cc:1035] T c9e60efa19a5402e82bf6a9ccc28efa5 P 23b67de5d91a47a782d84d24fef4b65e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c547b31ddca4bbd891d42f3ca09e0a1" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 35841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:30:01.038851 1592 mvcc.cc:205] Tried to move back new op lower bound from 7298858193040347136 to 7298858192966615040. Current Snapshot: MvccSnapshot[applied={T|T < 7298858193040347136}]
I20260620 09:30:01.039067 1594 mvcc.cc:205] Tried to move back new op lower bound from 7298858193040347136 to 7298858192966615040. Current Snapshot: MvccSnapshot[applied={T|T < 7298858193040347136}]
I20260620 09:30:01.039233 1593 mvcc.cc:205] Tried to move back new op lower bound from 7298858193040347136 to 7298858192966615040. Current Snapshot: MvccSnapshot[applied={T|T < 7298858193040347136}]
I20260620 09:30:01.041312 1103 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38524:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:30:01.041415 1103 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38524:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:30:01.042160 1103 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 04549979371e455e953c09ec661ccecc: Sending DeleteTablet for 3 replicas of tablet c9e60efa19a5402e82bf6a9ccc28efa5
I20260620 09:30:01.042459 1228 tablet_service.cc:1514] Processing DeleteTablet for tablet c9e60efa19a5402e82bf6a9ccc28efa5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:30:01 UTC) from {username='kudu', principal='kudu/127.16.43.190@KRBTEST.COM'} at 127.0.0.1:44572
I20260620 09:30:01.042500 1362 tablet_service.cc:1514] Processing DeleteTablet for tablet c9e60efa19a5402e82bf6a9ccc28efa5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:30:01 UTC) from {username='kudu', principal='kudu/127.16.43.190@KRBTEST.COM'} at 127.0.0.1:51348
I20260620 09:30:01.042508 1496 tablet_service.cc:1514] Processing DeleteTablet for tablet c9e60efa19a5402e82bf6a9ccc28efa5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:30:01 UTC) from {username='kudu', principal='kudu/127.16.43.190@KRBTEST.COM'} at 127.0.0.1:47868
I20260620 09:30:01.042748 1604 tablet_replica.cc:330] T c9e60efa19a5402e82bf6a9ccc28efa5 P 23b67de5d91a47a782d84d24fef4b65e: stopping tablet replica
I20260620 09:30:01.042795 1602 tablet_replica.cc:330] T c9e60efa19a5402e82bf6a9ccc28efa5 P 5537713527764d9590472593f39e4f02: stopping tablet replica
I20260620 09:30:01.042879 1604 raft_consensus.cc:2229] T c9e60efa19a5402e82bf6a9ccc28efa5 P 23b67de5d91a47a782d84d24fef4b65e [term 1 LEADER]: Raft consensus shutting down.
I20260620 09:30:01.042892 1602 raft_consensus.cc:2229] T c9e60efa19a5402e82bf6a9ccc28efa5 P 5537713527764d9590472593f39e4f02 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:30:01.042996 1602 raft_consensus.cc:2258] T c9e60efa19a5402e82bf6a9ccc28efa5 P 5537713527764d9590472593f39e4f02 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:30:01.043000 1604 raft_consensus.cc:2258] T c9e60efa19a5402e82bf6a9ccc28efa5 P 23b67de5d91a47a782d84d24fef4b65e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:30:01.043190 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 1165
I20260620 09:30:01.043942 1604 ts_tablet_manager.cc:1905] T c9e60efa19a5402e82bf6a9ccc28efa5 P 23b67de5d91a47a782d84d24fef4b65e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:30:01.046010 1604 ts_tablet_manager.cc:1918] T c9e60efa19a5402e82bf6a9ccc28efa5 P 23b67de5d91a47a782d84d24fef4b65e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:30:01.046075 1604 log.cc:1196] T c9e60efa19a5402e82bf6a9ccc28efa5 P 23b67de5d91a47a782d84d24fef4b65e: Deleting WAL directory at /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781947562949448-16558-0/minicluster-data/ts-2/wal/wals/c9e60efa19a5402e82bf6a9ccc28efa5
I20260620 09:30:01.046339 1604 ts_tablet_manager.cc:1939] T c9e60efa19a5402e82bf6a9ccc28efa5 P 23b67de5d91a47a782d84d24fef4b65e: Deleting consensus metadata
I20260620 09:30:01.046844 1088 catalog_manager.cc:4841] TS 23b67de5d91a47a782d84d24fef4b65e (127.16.43.131:41995): tablet c9e60efa19a5402e82bf6a9ccc28efa5 (table test-table [id=a4614f011b894b8899840ee4a71bd0c9]) successfully deleted
W20260620 09:30:01.051705 1087 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.43.129:40055 (error 108)
I20260620 09:30:01.051767 1603 tablet_replica.cc:330] T c9e60efa19a5402e82bf6a9ccc28efa5 P 8c547b31ddca4bbd891d42f3ca09e0a1: stopping tablet replica
I20260620 09:30:01.051847 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 1299
I20260620 09:30:01.051887 1603 raft_consensus.cc:2229] T c9e60efa19a5402e82bf6a9ccc28efa5 P 8c547b31ddca4bbd891d42f3ca09e0a1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:30:01.052004 1603 raft_consensus.cc:2258] T c9e60efa19a5402e82bf6a9ccc28efa5 P 8c547b31ddca4bbd891d42f3ca09e0a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:30:01.053267 1573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:01.053138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.129:40055 (local address 127.0.0.1:44586)
0620 09:30:01.053186 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0620 09:30:01.053223 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.129:40055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260620 09:30:01.053450 1087 catalog_manager.cc:4584] TS 5537713527764d9590472593f39e4f02 (127.16.43.129:40055): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c9e60efa19a5402e82bf6a9ccc28efa5: Network error: Client connection negotiation failed: client connection to 127.16.43.129:40055: connect: Connection refused (error 111)
I20260620 09:30:01.057220 1573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:01.057055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.130:35841 (local address 127.0.0.1:51352)
0620 09:30:01.057143 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0620 09:30:01.057175 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.130:35841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260620 09:30:01.057265 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 1433
I20260620 09:30:01.062259 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 1071
2026-06-20T09:30:01Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2192 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-20T09:30:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:30:01Z Disabled control of system clock
I20260620 09:30:01.090472 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781947562949448-16558-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781947562949448-16558-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781947562949448-16558-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.43.190:36081
--webserver_interface=127.16.43.190
--webserver_port=0
--builtin_ntp_servers=127.16.43.148:45999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.43.190:36081
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 09:30:01.172062 1609 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:30:01.172307 1609 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:30:01.172329 1609 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:30:01.173563 1609 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:30:01.173610 1609 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:30:01.173624 1609 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:30:01.173637 1609 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:30:01.173681 1609 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:30:01.175201 1609 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:45999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781947562949448-16558-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781947562949448-16558-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.43.190:36081
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.43.190:36081
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.43.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1609
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781947562949448-16558-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:30:01.175498 1609 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:30:01.175858 1609 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:30:01.180979 1615 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:30:01.180951 1617 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:30:01.181141 1609 server_base.cc:1034] running on GCE node
W20260620 09:30:01.180949 1614 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:30:01.181484 1609 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:30:01.181802 1609 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:30:01.182971 1609 hybrid_clock.cc:648] HybridClock initialized: now 1781947801182951 us; error 34 us; skew 500 ppm
I20260620 09:30:01.184361 1609 webserver.cc:460] Webserver started at http://127.16.43.190:44173/ using document root <none> and password file <none>
I20260620 09:30:01.184646 1609 fs_manager.cc:359] Metadata directory not provided
I20260620 09:30:01.184758 1609 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:30:01.184904 1609 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:30:01.186014 1609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781947562949448-16558-0/minicluster-data/master-0/data/instance:
uuid: "a8b6d85b5b834a2997399e19bc49f4be"
format_stamp: "Formatted at 2026-06-20 09:30:01 on dist-test-slave-706q"
server_key: "1478c3ce70383776d2a408b84982472d"
server_key_iv: "f1efc83a7890af63d0042f590d818163"
server_key_version: "encryptionkey@0"
I20260620 09:30:01.186383 1609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781947562949448-16558-0/minicluster-data/master-0/wal/instance:
uuid: "a8b6d85b5b834a2997399e19bc49f4be"
format_stamp: "Formatted at 2026-06-20 09:30:01 on dist-test-slave-706q"
server_key: "1478c3ce70383776d2a408b84982472d"
server_key_iv: "f1efc83a7890af63d0042f590d818163"
server_key_version: "encryptionkey@0"
I20260620 09:30:01.188757 1609 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.002s
I20260620 09:30:01.189538 1623 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:30:01.189733 1609 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 09:30:01.189834 1609 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781947562949448-16558-0/minicluster-data/master-0/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781947562949448-16558-0/minicluster-data/master-0/wal
uuid: "a8b6d85b5b834a2997399e19bc49f4be"
format_stamp: "Formatted at 2026-06-20 09:30:01 on dist-test-slave-706q"
server_key: "1478c3ce70383776d2a408b84982472d"
server_key_iv: "f1efc83a7890af63d0042f590d818163"
server_key_version: "encryptionkey@0"
I20260620 09:30:01.189944 1609 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781947562949448-16558-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781947562949448-16558-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781947562949448-16558-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:30:01.211896 1609 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:30:01.212251 1609 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:30:01.212447 1609 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:30:01.217010 1609 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.190:36081
I20260620 09:30:01.217167 1675 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.190:36081 every 8 connection(s)
I20260620 09:30:01.217396 1609 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
I20260620 09:30:01.218360 1676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:30:01.220701 1676 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8b6d85b5b834a2997399e19bc49f4be: Bootstrap starting.
I20260620 09:30:01.221299 1676 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a8b6d85b5b834a2997399e19bc49f4be: Neither blocks nor log segments found. Creating new log.
I20260620 09:30:01.221601 1676 log.cc:824] T 00000000000000000000000000000000 P a8b6d85b5b834a2997399e19bc49f4be: Log is configured to *not* fsync() on all Append() calls
I20260620 09:30:01.222298 1676 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8b6d85b5b834a2997399e19bc49f4be: No bootstrap required, opened a new log
I20260620 09:30:01.224058 1676 raft_consensus.cc:348] T 00000000000000000000000000000000 P a8b6d85b5b834a2997399e19bc49f4be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8b6d85b5b834a2997399e19bc49f4be" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 36081 } }
I20260620 09:30:01.224207 1676 raft_consensus.cc:374] T 00000000000000000000000000000000 P a8b6d85b5b834a2997399e19bc49f4be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:30:01.224231 1676 raft_consensus.cc:729] T 00000000000000000000000000000000 P a8b6d85b5b834a2997399e19bc49f4be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8b6d85b5b834a2997399e19bc49f4be, State: Initialized, Role: FOLLOWER
I20260620 09:30:01.224409 1676 consensus_queue.cc:260] T 00000000000000000000000000000000 P a8b6d85b5b834a2997399e19bc49f4be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8b6d85b5b834a2997399e19bc49f4be" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 36081 } }
I20260620 09:30:01.224472 1676 raft_consensus.cc:388] T 00000000000000000000000000000000 P a8b6d85b5b834a2997399e19bc49f4be [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:30:01.224512 1676 raft_consensus.cc:482] T 00000000000000000000000000000000 P a8b6d85b5b834a2997399e19bc49f4be [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:30:01.224562 1676 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a8b6d85b5b834a2997399e19bc49f4be [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:30:01.225107 1676 raft_consensus.cc:504] T 00000000000000000000000000000000 P a8b6d85b5b834a2997399e19bc49f4be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8b6d85b5b834a2997399e19bc49f4be" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 36081 } }
I20260620 09:30:01.225198 1676 leader_election.cc:302] T 00000000000000000000000000000000 P a8b6d85b5b834a2997399e19bc49f4be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a8b6d85b5b834a2997399e19bc49f4be; no voters:
I20260620 09:30:01.225401 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 1609
I20260620 09:30:01.225412 1676 leader_election.cc:288] T 00000000000000000000000000000000 P a8b6d85b5b834a2997399e19bc49f4be [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:30:01.225450 1679 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a8b6d85b5b834a2997399e19bc49f4be [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:30:01.225528 16558 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781947562949448-16558-0/minicluster-data/master-0/wal/instance
I20260620 09:30:01.225574 1679 raft_consensus.cc:686] T 00000000000000000000000000000000 P a8b6d85b5b834a2997399e19bc49f4be [term 1 LEADER]: Becoming Leader. State: Replica: a8b6d85b5b834a2997399e19bc49f4be, State: Running, Role: LEADER
I20260620 09:30:01.225669 1679 consensus_queue.cc:237] T 00000000000000000000000000000000 P a8b6d85b5b834a2997399e19bc49f4be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8b6d85b5b834a2997399e19bc49f4be" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 36081 } }
I20260620 09:30:01.225701 1676 sys_catalog.cc:564] T 00000000000000000000000000000000 P a8b6d85b5b834a2997399e19bc49f4be [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:30:01.225728 16558 external_mini_cluster.cc:1428] Setting key 3e52e9e45a121d5cf88e229263a86d07
I20260620 09:30:01.225973 1679 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8b6d85b5b834a2997399e19bc49f4be [sys.catalog]: SysCatalogTable state changed. Reason: New leader a8b6d85b5b834a2997399e19bc49f4be. Latest consensus state: current_term: 1 leader_uuid: "a8b6d85b5b834a2997399e19bc49f4be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8b6d85b5b834a2997399e19bc49f4be" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 36081 } } }
I20260620 09:30:01.226152 1679 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8b6d85b5b834a2997399e19bc49f4be [sys.catalog]: This master's current role is: LEADER
I20260620 09:30:01.226923 1680 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8b6d85b5b834a2997399e19bc49f4be [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a8b6d85b5b834a2997399e19bc49f4be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8b6d85b5b834a2997399e19bc49f4be" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 36081 } } }
I20260620 09:30:01.227010 1680 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8b6d85b5b834a2997399e19bc49f4be [sys.catalog]: This master's current role is: LEADER
I20260620 09:30:01.226431 1688 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:30:01.227521 1688 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:30:01.229630 1688 catalog_manager.cc:1347] Generated new cluster ID: 5f107b1f4e9b4f86aeaf1ac5cbfa3c6c
I20260620 09:30:01.229692 1688 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:30:01.232301 1690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:01.226385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44250 (local address 127.16.43.190:36081)
0620 09:30:01.226694 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:01.226729 (+ 35us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:01.226997 (+ 268us) server_negotiation.cc:407] Connection header received
0620 09:30:01.227161 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:01.227168 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:01.227246 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:01.227400 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:30:01.227758 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:01.229110 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:01.230285 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:01.230741 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:01.230882 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:01.231644 (+ 762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:30:01.231648 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:30:01.231648 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:30:01.231828 (+ 180us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:30:01.231861 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:30:01.231873 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:30:01.231978 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:01.232048 (+ 70us) server_negotiation.cc:299] Negotiation successful
0620 09:30:01.232129 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":83,"threads_started":1}
I20260620 09:30:01.235091 1688 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:30:01.235680 1688 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:30:01.240677 1688 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a8b6d85b5b834a2997399e19bc49f4be: Generated new TSK 0
I20260620 09:30:01.240883 1688 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 09:30:01.244113 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781947562949448-16558-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781947562949448-16558-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781947562949448-16558-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781947562949448-16558-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.43.129:0
--local_ip_for_outbound_sockets=127.16.43.129
--webserver_interface=127.16.43.129
--webserver_port=0
--tserver_master_addrs=127.16.43.190:36081
--builtin_ntp_servers=127.16.43.148:45999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 09:30:01.344687 1700 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:30:01.344944 1700 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:30:01.344982 1700 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:30:01.346158 1700 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:30:01.346226 1700 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:30:01.346268 1700 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.43.129
I20260620 09:30:01.347707 1700 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:45999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781947562949448-16558-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781947562949448-16558-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.43.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781947562949448-16558-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.43.129
--webserver_port=0
--tserver_master_addrs=127.16.43.190:36081
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1700
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.43.129
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781947562949448-16558-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:30:01.347975 1700 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:30:01.348304 1700 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:30:01.353600 1705 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:30:01.353613 1706 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:30:01.353705 1708 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:30:01.354107 1700 server_base.cc:1034] running on GCE node
I20260620 09:30:01.354326 1700 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:30:01.354588 1700 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:30:01.355751 1700 hybrid_clock.cc:648] HybridClock initialized: now 1781947801355727 us; error 45 us; skew 500 ppm
I20260620 09:30:01.357089 1700 webserver.cc:460] Webserver started at http://127.16.43.129:43791/ using document root <none> and password file <none>
I20260620 09:30:01.357362 1700 fs_manager.cc:359] Metadata directory not provided
I20260620 09:30:01.357430 1700 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:30:01.357628 1700 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:30:01.358723 1700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781947562949448-16558-0/minicluster-data/ts-0/data/instance:
uuid: "6d15d5c052284c499e2fb50845fc1926"
format_stamp: "Formatted at 2026-06-20 09:30:01 on dist-test-slave-706q"
server_key: "16e3b590d2523c168c16fb9817d3c72c"
server_key_iv: "c6758f8ba54b02784b6ee3ba4bd3c30b"
server_key_version: "encryptionkey@0"
I20260620 09:30:01.359098 1700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781947562949448-16558-0/minicluster-data/ts-0/wal/instance:
uuid: "6d15d5c052284c499e2fb50845fc1926"
format_stamp: "Formatted at 2026-06-20 09:30:01 on dist-test-slave-706q"
server_key: "16e3b590d2523c168c16fb9817d3c72c"
server_key_iv: "c6758f8ba54b02784b6ee3ba4bd3c30b"
server_key_version: "encryptionkey@0"
I20260620 09:30:01.360579 1700 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 09:30:01.361373 1714 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:30:01.361611 1700 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 09:30:01.361716 1700 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781947562949448-16558-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781947562949448-16558-0/minicluster-data/ts-0/wal
uuid: "6d15d5c052284c499e2fb50845fc1926"
format_stamp: "Formatted at 2026-06-20 09:30:01 on dist-test-slave-706q"
server_key: "16e3b590d2523c168c16fb9817d3c72c"
server_key_iv: "c6758f8ba54b02784b6ee3ba4bd3c30b"
server_key_version: "encryptionkey@0"
I20260620 09:30:01.361797 1700 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781947562949448-16558-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781947562949448-16558-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781947562949448-16558-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:30:01.380534 1700 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:30:01.380977 1700 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:30:01.381130 1700 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:30:01.381394 1700 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:30:01.381747 1700 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:30:01.381829 1700 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:30:01.381888 1700 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:30:01.381922 1700 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:30:01.389500 1700 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.129:42071
I20260620 09:30:01.389542 1827 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.129:42071 every 8 connection(s)
I20260620 09:30:01.389959 1700 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781947562949448-16558-0/minicluster-data/ts-0/data/info.pb
I20260620 09:30:01.390175 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 1700
I20260620 09:30:01.390280 16558 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781947562949448-16558-0/minicluster-data/ts-0/wal/instance
I20260620 09:30:01.390506 16558 external_mini_cluster.cc:1428] Setting key 3cc99fbaf878163ca63cd1b23df9ed06
I20260620 09:30:01.391669 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781947562949448-16558-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781947562949448-16558-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781947562949448-16558-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781947562949448-16558-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.43.130:0
--local_ip_for_outbound_sockets=127.16.43.130
--webserver_interface=127.16.43.130
--webserver_port=0
--tserver_master_addrs=127.16.43.190:36081
--builtin_ntp_servers=127.16.43.148:45999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 09:30:01.396410 1690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:01.391044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.129:53865 (local address 127.16.43.190:36081)
0620 09:30:01.391144 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:01.391146 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:01.391272 (+ 126us) server_negotiation.cc:407] Connection header received
0620 09:30:01.391978 (+ 706us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:01.391979 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:01.391993 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:01.392066 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:30:01.392955 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:01.393614 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:01.394732 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:01.394978 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:01.395074 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:01.396126 (+ 1052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:30:01.396127 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:30:01.396128 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:30:01.396133 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:30:01.396150 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:30:01.396154 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:30:01.396208 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:01.396289 (+ 81us) server_negotiation.cc:299] Negotiation successful
0620 09:30:01.396323 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 09:30:01.396651 1830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:01.390780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:36081 (local address 127.16.43.129:53865)
0620 09:30:01.391141 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0620 09:30:01.391194 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 09:30:01.391883 (+ 689us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:30:01.392171 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:30:01.392175 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:30:01.392187 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:30:01.392825 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:30:01.392831 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:30:01.393718 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:30:01.393720 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:30:01.394613 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:30:01.394617 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:30:01.395087 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:30:01.395088 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:30:01.395237 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:01.395939 (+ 702us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 09:30:01.395939 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:30:01.395969 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 09:30:01.396001 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:30:01.396218 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:30:01.396223 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:30:01.396226 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 09:30:01.396292 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 09:30:01.396402 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":90,"threads_started":1}
I20260620 09:30:01.397265 1828 heartbeater.cc:347] Connected to a master server at 127.16.43.190:36081
I20260620 09:30:01.397396 1828 heartbeater.cc:464] Registering TS with master...
I20260620 09:30:01.397598 1828 heartbeater.cc:511] Master 127.16.43.190:36081 requested a full tablet report, sending...
I20260620 09:30:01.398119 1640 ts_manager.cc:194] Registered new tserver with Master: 6d15d5c052284c499e2fb50845fc1926 (127.16.43.129:42071)
I20260620 09:30:01.398778 1640 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.43.129:53865
W20260620 09:30:01.475732 1831 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:30:01.475939 1831 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:30:01.475962 1831 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:30:01.477092 1831 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:30:01.477128 1831 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:30:01.477152 1831 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.43.130
I20260620 09:30:01.478444 1831 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:45999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781947562949448-16558-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781947562949448-16558-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.43.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781947562949448-16558-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.43.130
--webserver_port=0
--tserver_master_addrs=127.16.43.190:36081
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1831
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.43.130
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781947562949448-16558-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:30:01.478650 1831 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:30:01.478895 1831 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:30:01.484241 1839 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:30:01.484277 1837 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:30:01.484241 1836 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:30:01.484342 1831 server_base.cc:1034] running on GCE node
I20260620 09:30:01.484747 1831 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:30:01.485013 1831 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:30:01.486207 1831 hybrid_clock.cc:648] HybridClock initialized: now 1781947801486190 us; error 31 us; skew 500 ppm
I20260620 09:30:01.487632 1831 webserver.cc:460] Webserver started at http://127.16.43.130:35327/ using document root <none> and password file <none>
I20260620 09:30:01.487905 1831 fs_manager.cc:359] Metadata directory not provided
I20260620 09:30:01.487977 1831 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:30:01.488131 1831 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:30:01.489244 1831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781947562949448-16558-0/minicluster-data/ts-1/data/instance:
uuid: "a8116ad43adb41e298726dfee1f24a8a"
format_stamp: "Formatted at 2026-06-20 09:30:01 on dist-test-slave-706q"
server_key: "663ee163017fc4dfb4b69a1e75bf705b"
server_key_iv: "79ec2c8fe52d9ecbbb96b741c5cda579"
server_key_version: "encryptionkey@0"
I20260620 09:30:01.489655 1831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781947562949448-16558-0/minicluster-data/ts-1/wal/instance:
uuid: "a8116ad43adb41e298726dfee1f24a8a"
format_stamp: "Formatted at 2026-06-20 09:30:01 on dist-test-slave-706q"
server_key: "663ee163017fc4dfb4b69a1e75bf705b"
server_key_iv: "79ec2c8fe52d9ecbbb96b741c5cda579"
server_key_version: "encryptionkey@0"
I20260620 09:30:01.491034 1831 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 09:30:01.491998 1845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:30:01.492378 1831 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 09:30:01.492489 1831 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781947562949448-16558-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781947562949448-16558-0/minicluster-data/ts-1/wal
uuid: "a8116ad43adb41e298726dfee1f24a8a"
format_stamp: "Formatted at 2026-06-20 09:30:01 on dist-test-slave-706q"
server_key: "663ee163017fc4dfb4b69a1e75bf705b"
server_key_iv: "79ec2c8fe52d9ecbbb96b741c5cda579"
server_key_version: "encryptionkey@0"
I20260620 09:30:01.492578 1831 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781947562949448-16558-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781947562949448-16558-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781947562949448-16558-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:30:01.501422 1831 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:30:01.501758 1831 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:30:01.501917 1831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:30:01.502161 1831 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:30:01.502492 1831 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:30:01.502548 1831 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:30:01.502585 1831 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:30:01.502633 1831 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:30:01.508504 1831 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.130:46699
I20260620 09:30:01.508586 1958 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.130:46699 every 8 connection(s)
I20260620 09:30:01.508924 1831 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781947562949448-16558-0/minicluster-data/ts-1/data/info.pb
I20260620 09:30:01.514456 1690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:01.509757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.130:48679 (local address 127.16.43.190:36081)
0620 09:30:01.509831 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:01.509834 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:01.510019 (+ 185us) server_negotiation.cc:407] Connection header received
0620 09:30:01.510579 (+ 560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:01.510581 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:01.510594 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:01.510659 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:30:01.511365 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:01.512000 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:01.513013 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:01.513207 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:01.513284 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:01.514122 (+ 838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:30:01.514123 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:30:01.514124 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:30:01.514129 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:30:01.514146 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:30:01.514150 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:30:01.514226 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:01.514335 (+ 109us) server_negotiation.cc:299] Negotiation successful
0620 09:30:01.514371 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 09:30:01.514683 1961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:01.509646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:36081 (local address 127.16.43.130:48679)
0620 09:30:01.509885 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0620 09:30:01.509932 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 09:30:01.510462 (+ 530us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:30:01.510728 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:30:01.510732 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:30:01.510742 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:30:01.511258 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:30:01.511262 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:30:01.512114 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:30:01.512115 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:30:01.512907 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:30:01.512909 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:30:01.513291 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:30:01.513292 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:30:01.513436 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:01.513947 (+ 511us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 09:30:01.513947 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:30:01.513977 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 09:30:01.513999 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:30:01.514228 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:30:01.514235 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:30:01.514238 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 09:30:01.514338 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 09:30:01.514448 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":111,"threads_started":1}
I20260620 09:30:01.515187 1959 heartbeater.cc:347] Connected to a master server at 127.16.43.190:36081
I20260620 09:30:01.515290 1959 heartbeater.cc:464] Registering TS with master...
I20260620 09:30:01.515465 1959 heartbeater.cc:511] Master 127.16.43.190:36081 requested a full tablet report, sending...
I20260620 09:30:01.516000 1640 ts_manager.cc:194] Registered new tserver with Master: a8116ad43adb41e298726dfee1f24a8a (127.16.43.130:46699)
I20260620 09:30:01.516434 1640 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.43.130:48679
I20260620 09:30:01.517293 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 1831
I20260620 09:30:01.517371 16558 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781947562949448-16558-0/minicluster-data/ts-1/wal/instance
I20260620 09:30:01.517609 16558 external_mini_cluster.cc:1428] Setting key 4c14cb492b55eef59e9cb0345f955a71
I20260620 09:30:01.518541 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781947562949448-16558-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781947562949448-16558-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781947562949448-16558-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781947562949448-16558-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.43.131:0
--local_ip_for_outbound_sockets=127.16.43.131
--webserver_interface=127.16.43.131
--webserver_port=0
--tserver_master_addrs=127.16.43.190:36081
--builtin_ntp_servers=127.16.43.148:45999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 09:30:01.602813 1962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:30:01.603055 1962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:30:01.603080 1962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:30:01.604437 1962 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:30:01.604496 1962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:30:01.604526 1962 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.43.131
I20260620 09:30:01.605990 1962 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:45999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781947562949448-16558-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781947562949448-16558-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.43.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781947562949448-16558-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.43.131
--webserver_port=0
--tserver_master_addrs=127.16.43.190:36081
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1962
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.43.131
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781947562949448-16558-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:30:01.606292 1962 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:30:01.606632 1962 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:30:01.611955 1970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:30:01.611963 1967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:30:01.612046 1968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:30:01.612176 1962 server_base.cc:1034] running on GCE node
I20260620 09:30:01.612450 1962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:30:01.612659 1962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:30:01.613832 1962 hybrid_clock.cc:648] HybridClock initialized: now 1781947801613770 us; error 42 us; skew 500 ppm
I20260620 09:30:01.615218 1962 webserver.cc:460] Webserver started at http://127.16.43.131:45555/ using document root <none> and password file <none>
I20260620 09:30:01.615430 1962 fs_manager.cc:359] Metadata directory not provided
I20260620 09:30:01.615474 1962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:30:01.615672 1962 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:30:01.616814 1962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781947562949448-16558-0/minicluster-data/ts-2/data/instance:
uuid: "b62ec926ff704888a2b357bbb96b8ddf"
format_stamp: "Formatted at 2026-06-20 09:30:01 on dist-test-slave-706q"
server_key: "873037d957e288ee3a9db5714f879257"
server_key_iv: "0a324d9aa61c9e00eb3757851b6cd021"
server_key_version: "encryptionkey@0"
I20260620 09:30:01.617134 1962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781947562949448-16558-0/minicluster-data/ts-2/wal/instance:
uuid: "b62ec926ff704888a2b357bbb96b8ddf"
format_stamp: "Formatted at 2026-06-20 09:30:01 on dist-test-slave-706q"
server_key: "873037d957e288ee3a9db5714f879257"
server_key_iv: "0a324d9aa61c9e00eb3757851b6cd021"
server_key_version: "encryptionkey@0"
I20260620 09:30:01.618453 1962 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 09:30:01.619270 1976 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:30:01.619485 1962 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:30:01.619659 1962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781947562949448-16558-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781947562949448-16558-0/minicluster-data/ts-2/wal
uuid: "b62ec926ff704888a2b357bbb96b8ddf"
format_stamp: "Formatted at 2026-06-20 09:30:01 on dist-test-slave-706q"
server_key: "873037d957e288ee3a9db5714f879257"
server_key_iv: "0a324d9aa61c9e00eb3757851b6cd021"
server_key_version: "encryptionkey@0"
I20260620 09:30:01.619747 1962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781947562949448-16558-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781947562949448-16558-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781947562949448-16558-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:30:01.672454 1962 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:30:01.672855 1962 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:30:01.673027 1962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:30:01.673291 1962 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:30:01.673728 1962 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:30:01.673804 1962 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:30:01.673848 1962 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:30:01.673877 1962 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:30:01.679767 1962 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.131:46185
I20260620 09:30:01.679812 2089 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.131:46185 every 8 connection(s)
I20260620 09:30:01.680164 1962 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781947562949448-16558-0/minicluster-data/ts-2/data/info.pb
I20260620 09:30:01.684613 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 1962
I20260620 09:30:01.684731 16558 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781947562949448-16558-0/minicluster-data/ts-2/wal/instance
I20260620 09:30:01.684932 16558 external_mini_cluster.cc:1428] Setting key ad1a1df37dc8a2c410b79f5b65adb87d
I20260620 09:30:01.686033 1690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:01.681036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.131:48595 (local address 127.16.43.190:36081)
0620 09:30:01.681104 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:01.681106 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:01.681372 (+ 266us) server_negotiation.cc:407] Connection header received
0620 09:30:01.681865 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:01.681867 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:01.681879 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:01.681947 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:30:01.682675 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:01.683246 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:01.684339 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:01.684543 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:01.684597 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:01.685732 (+ 1135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:30:01.685733 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:30:01.685734 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:30:01.685739 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:30:01.685756 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:30:01.685760 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:30:01.685828 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:01.685922 (+ 94us) server_negotiation.cc:299] Negotiation successful
0620 09:30:01.685953 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 09:30:01.686250 2092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:01.681007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:36081 (local address 127.16.43.131:48595)
0620 09:30:01.681231 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0620 09:30:01.681274 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 09:30:01.681769 (+ 495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:30:01.682027 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:30:01.682030 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:30:01.682039 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:30:01.682563 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:30:01.682566 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:30:01.683355 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:30:01.683356 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:30:01.684258 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:30:01.684262 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:30:01.684638 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:30:01.684639 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:30:01.684812 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:01.685502 (+ 690us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 09:30:01.685502 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:30:01.685533 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 09:30:01.685558 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:30:01.685842 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:30:01.685848 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:30:01.685850 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 09:30:01.685921 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 09:30:01.686028 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":121,"threads_started":1}
I20260620 09:30:01.686864 2090 heartbeater.cc:347] Connected to a master server at 127.16.43.190:36081
I20260620 09:30:01.686995 2090 heartbeater.cc:464] Registering TS with master...
I20260620 09:30:01.687186 2090 heartbeater.cc:511] Master 127.16.43.190:36081 requested a full tablet report, sending...
I20260620 09:30:01.687624 1640 ts_manager.cc:194] Registered new tserver with Master: b62ec926ff704888a2b357bbb96b8ddf (127.16.43.131:46185)
I20260620 09:30:01.688062 1640 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.43.131:48595
I20260620 09:30:01.697254 16558 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 09:30:01.702311 1690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:01.699093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44256 (local address 127.16.43.190:36081)
0620 09:30:01.699169 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:01.699172 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:01.699272 (+ 100us) server_negotiation.cc:407] Connection header received
0620 09:30:01.699336 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:01.699337 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:01.699349 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:01.699393 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:30:01.699580 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:01.700118 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:01.700932 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:01.701136 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:01.701207 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:01.701964 (+ 757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:30:01.701965 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:30:01.701966 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:30:01.701971 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:30:01.701986 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:30:01.701990 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:30:01.702037 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:01.702154 (+ 117us) server_negotiation.cc:299] Negotiation successful
0620 09:30:01.702181 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 09:30:01.704319 1640 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44256:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 09:30:01.706012 1640 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 09:30:01.713160 2100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:01.708517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58948 (local address 127.16.43.129:42071)
0620 09:30:01.708739 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:01.708745 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:01.709395 (+ 650us) server_negotiation.cc:407] Connection header received
0620 09:30:01.709486 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:01.709489 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:01.709505 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:01.709659 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:30:01.709790 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:01.710816 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:01.712023 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:01.712713 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:01.712793 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:01.712888 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:01.713022 (+ 134us) server_negotiation.cc:299] Negotiation successful
0620 09:30:01.713064 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":103,"threads_started":1}
I20260620 09:30:01.713158 2099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:01.708440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.129:42071 (local address 127.0.0.1:58948)
0620 09:30:01.709327 (+ 887us) negotiation.cc:107] Waiting for socket to connect
0620 09:30:01.709339 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 09:30:01.709402 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:30:01.709668 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:30:01.709668 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:30:01.709670 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:30:01.709730 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:30:01.709733 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:30:01.710906 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:30:01.710907 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:30:01.711901 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:30:01.711904 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:30:01.712811 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:30:01.712813 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:30:01.712901 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:01.712916 (+ 15us) client_negotiation.cc:769] Sending connection context
0620 09:30:01.712999 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 09:30:01.713060 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":872,"mutex_wait_us":45,"thread_start_us":75,"threads_started":1}
I20260620 09:30:01.713713 2098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:01.708675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.131:46185 (local address 127.0.0.1:38888)
0620 09:30:01.709501 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0620 09:30:01.709510 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 09:30:01.709556 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:30:01.709814 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:30:01.709815 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:30:01.709816 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:30:01.709846 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:30:01.709847 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:30:01.710905 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:30:01.710906 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:30:01.712324 (+ 1418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:30:01.712327 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:30:01.713405 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:30:01.713406 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:30:01.713497 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:01.713502 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 09:30:01.713559 (+ 57us) client_negotiation.cc:240] Negotiation successful
0620 09:30:01.713608 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":814,"thread_start_us":88,"threads_started":1}
I20260620 09:30:01.713873 2103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:01.708840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49676 (local address 127.16.43.130:46699)
0620 09:30:01.709037 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:01.709041 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:01.709051 (+ 10us) server_negotiation.cc:407] Connection header received
0620 09:30:01.709087 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:01.709089 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:01.709102 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:01.709228 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:30:01.709425 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:01.710587 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:01.712551 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:01.713308 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:01.713386 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:01.713475 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:01.713755 (+ 280us) server_negotiation.cc:299] Negotiation successful
0620 09:30:01.713793 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":78,"threads_started":1}
I20260620 09:30:01.713995 2101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:01.708431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.130:46699 (local address 127.0.0.1:49676)
0620 09:30:01.708817 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0620 09:30:01.708848 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0620 09:30:01.708942 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:30:01.709232 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:30:01.709236 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:30:01.709240 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:30:01.709322 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:30:01.709324 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:30:01.710691 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:30:01.710692 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:30:01.712459 (+ 1767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:30:01.712461 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:30:01.713515 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:30:01.713517 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:30:01.713581 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:01.713585 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 09:30:01.713637 (+ 52us) client_negotiation.cc:240] Negotiation successful
0620 09:30:01.713682 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"mutex_wait_us":41,"thread_start_us":89,"threads_started":1}
I20260620 09:30:01.714042 2102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:01.708789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38888 (local address 127.16.43.131:46185)
0620 09:30:01.708986 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:01.708990 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:01.709558 (+ 568us) server_negotiation.cc:407] Connection header received
0620 09:30:01.709633 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:01.709636 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:01.709662 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:01.709742 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:30:01.709934 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:01.710796 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:01.712431 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:01.713195 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:01.713269 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:01.713361 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:01.713889 (+ 528us) server_negotiation.cc:299] Negotiation successful
0620 09:30:01.713938 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":100,"threads_started":1}
I20260620 09:30:01.714316 1762 tablet_service.cc:1467] Processing CreateTablet for tablet 364f6dacb9fa43b8a96aea8a54e4a2c4 (DEFAULT_TABLE table=test-table [id=0d3460de014f44678bdd551f4d25bc8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:30:01.714634 1762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 364f6dacb9fa43b8a96aea8a54e4a2c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:30:01.714636 1893 tablet_service.cc:1467] Processing CreateTablet for tablet 364f6dacb9fa43b8a96aea8a54e4a2c4 (DEFAULT_TABLE table=test-table [id=0d3460de014f44678bdd551f4d25bc8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:30:01.714890 1893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 364f6dacb9fa43b8a96aea8a54e4a2c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:30:01.717599 2104 tablet_bootstrap.cc:492] T 364f6dacb9fa43b8a96aea8a54e4a2c4 P 6d15d5c052284c499e2fb50845fc1926: Bootstrap starting.
I20260620 09:30:01.718124 2105 tablet_bootstrap.cc:492] T 364f6dacb9fa43b8a96aea8a54e4a2c4 P a8116ad43adb41e298726dfee1f24a8a: Bootstrap starting.
I20260620 09:30:01.718376 2104 tablet_bootstrap.cc:654] T 364f6dacb9fa43b8a96aea8a54e4a2c4 P 6d15d5c052284c499e2fb50845fc1926: Neither blocks nor log segments found. Creating new log.
I20260620 09:30:01.718556 2024 tablet_service.cc:1467] Processing CreateTablet for tablet 364f6dacb9fa43b8a96aea8a54e4a2c4 (DEFAULT_TABLE table=test-table [id=0d3460de014f44678bdd551f4d25bc8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:30:01.718751 2104 log.cc:824] T 364f6dacb9fa43b8a96aea8a54e4a2c4 P 6d15d5c052284c499e2fb50845fc1926: Log is configured to *not* fsync() on all Append() calls
I20260620 09:30:01.718796 2105 tablet_bootstrap.cc:654] T 364f6dacb9fa43b8a96aea8a54e4a2c4 P a8116ad43adb41e298726dfee1f24a8a: Neither blocks nor log segments found. Creating new log.
I20260620 09:30:01.718948 2024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 364f6dacb9fa43b8a96aea8a54e4a2c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:30:01.719115 2105 log.cc:824] T 364f6dacb9fa43b8a96aea8a54e4a2c4 P a8116ad43adb41e298726dfee1f24a8a: Log is configured to *not* fsync() on all Append() calls
I20260620 09:30:01.719723 2104 tablet_bootstrap.cc:492] T 364f6dacb9fa43b8a96aea8a54e4a2c4 P 6d15d5c052284c499e2fb50845fc1926: No bootstrap required, opened a new log
I20260620 09:30:01.719802 2104 ts_tablet_manager.cc:1397] T 364f6dacb9fa43b8a96aea8a54e4a2c4 P 6d15d5c052284c499e2fb50845fc1926: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 09:30:01.720173 2105 tablet_bootstrap.cc:492] T 364f6dacb9fa43b8a96aea8a54e4a2c4 P a8116ad43adb41e298726dfee1f24a8a: No bootstrap required, opened a new log
I20260620 09:30:01.720242 2105 ts_tablet_manager.cc:1397] T 364f6dacb9fa43b8a96aea8a54e4a2c4 P a8116ad43adb41e298726dfee1f24a8a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 09:30:01.722187 2105 raft_consensus.cc:348] T 364f6dacb9fa43b8a96aea8a54e4a2c4 P a8116ad43adb41e298726dfee1f24a8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d15d5c052284c499e2fb50845fc1926" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 42071 } } peers { permanent_uuid: "a8116ad43adb41e298726dfee1f24a8a" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 46699 } } peers { permanent_uuid: "b62ec926ff704888a2b357bbb96b8ddf" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 46185 } }
I20260620 09:30:01.722373 2105 raft_consensus.cc:374] T 364f6dacb9fa43b8a96aea8a54e4a2c4 P a8116ad43adb41e298726dfee1f24a8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:30:01.722414 2105 raft_consensus.cc:729] T 364f6dacb9fa43b8a96aea8a54e4a2c4 P a8116ad43adb41e298726dfee1f24a8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8116ad43adb41e298726dfee1f24a8a, State: Initialized, Role: FOLLOWER
I20260620 09:30:01.722559 2105 consensus_queue.cc:260] T 364f6dacb9fa43b8a96aea8a54e4a2c4 P a8116ad43adb41e298726dfee1f24a8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d15d5c052284c499e2fb50845fc1926" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 42071 } } peers { permanent_uuid: "a8116ad43adb41e298726dfee1f24a8a" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 46699 } } peers { permanent_uuid: "b62ec926ff704888a2b357bbb96b8ddf" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 46185 } }
I20260620 09:30:01.722870 2105 ts_tablet_manager.cc:1428] T 364f6dacb9fa43b8a96aea8a54e4a2c4 P a8116ad43adb41e298726dfee1f24a8a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 09:30:01.722882 2108 tablet_bootstrap.cc:492] T 364f6dacb9fa43b8a96aea8a54e4a2c4 P b62ec926ff704888a2b357bbb96b8ddf: Bootstrap starting.
I20260620 09:30:01.722821 2104 raft_consensus.cc:348] T 364f6dacb9fa43b8a96aea8a54e4a2c4 P 6d15d5c052284c499e2fb50845fc1926 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d15d5c052284c499e2fb50845fc1926" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 42071 } } peers { permanent_uuid: "a8116ad43adb41e298726dfee1f24a8a" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 46699 } } peers { permanent_uuid: "b62ec926ff704888a2b357bbb96b8ddf" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 46185 } }
I20260620 09:30:01.723016 1959 heartbeater.cc:503] Master 127.16.43.190:36081 was elected leader, sending a full tablet report...
I20260620 09:30:01.722993 2104 raft_consensus.cc:374] T 364f6dacb9fa43b8a96aea8a54e4a2c4 P 6d15d5c052284c499e2fb50845fc1926 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:30:01.723063 2104 raft_consensus.cc:729] T 364f6dacb9fa43b8a96aea8a54e4a2c4 P 6d15d5c052284c499e2fb50845fc1926 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d15d5c052284c499e2fb50845fc1926, State: Initialized, Role: FOLLOWER
I20260620 09:30:01.723209 2104 consensus_queue.cc:260] T 364f6dacb9fa43b8a96aea8a54e4a2c4 P 6d15d5c052284c499e2fb50845fc1926 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d15d5c052284c499e2fb50845fc1926" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 42071 } } peers { permanent_uuid: "a8116ad43adb41e298726dfee1f24a8a" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 46699 } } peers { permanent_uuid: "b62ec926ff704888a2b357bbb96b8ddf" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 46185 } }
I20260620 09:30:01.723491 2104 ts_tablet_manager.cc:1428] T 364f6dacb9fa43b8a96aea8a54e4a2c4 P 6d15d5c052284c499e2fb50845fc1926: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260620 09:30:01.723667 1828 heartbeater.cc:503] Master 127.16.43.190:36081 was elected leader, sending a full tablet report...
I20260620 09:30:01.723708 2108 tablet_bootstrap.cc:654] T 364f6dacb9fa43b8a96aea8a54e4a2c4 P b62ec926ff704888a2b357bbb96b8ddf: Neither blocks nor log segments found. Creating new log.
I20260620 09:30:01.724051 2108 log.cc:824] T 364f6dacb9fa43b8a96aea8a54e4a2c4 P b62ec926ff704888a2b357bbb96b8ddf: Log is configured to *not* fsync() on all Append() calls
I20260620 09:30:01.724910 2108 tablet_bootstrap.cc:492] T 364f6dacb9fa43b8a96aea8a54e4a2c4 P b62ec926ff704888a2b357bbb96b8ddf: No bootstrap required, opened a new log
I20260620 09:30:01.724962 2108 ts_tablet_manager.cc:1397] T 364f6dacb9fa43b8a96aea8a54e4a2c4 P b62ec926ff704888a2b357bbb96b8ddf: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 09:30:01.727084 2108 raft_consensus.cc:348] T 364f6dacb9fa43b8a96aea8a54e4a2c4 P b62ec926ff704888a2b357bbb96b8ddf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d15d5c052284c499e2fb50845fc1926" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 42071 } } peers { permanent_uuid: "a8116ad43adb41e298726dfee1f24a8a" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 46699 } } peers { permanent_uuid: "b62ec926ff704888a2b357bbb96b8ddf" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 46185 } }
I20260620 09:30:01.727265 2108 raft_consensus.cc:374] T 364f6dacb9fa43b8a96aea8a54e4a2c4 P b62ec926ff704888a2b357bbb96b8ddf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:30:01.727299 2108 raft_consensus.cc:729] T 364f6dacb9fa43b8a96aea8a54e4a2c4 P b62ec926ff704888a2b357bbb96b8ddf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b62ec926ff704888a2b357bbb96b8ddf, State: Initialized, Role: FOLLOWER
I20260620 09:30:01.727414 2108 consensus_queue.cc:260] T 364f6dacb9fa43b8a96aea8a54e4a2c4 P b62ec926ff704888a2b357bbb96b8ddf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d15d5c052284c499e2fb50845fc1926" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 42071 } } peers { permanent_uuid: "a8116ad43adb41e298726dfee1f24a8a" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 46699 } } peers { permanent_uuid: "b62ec926ff704888a2b357bbb96b8ddf" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 46185 } }
I20260620 09:30:01.727696 2108 ts_tablet_manager.cc:1428] T 364f6dacb9fa43b8a96aea8a54e4a2c4 P b62ec926ff704888a2b357bbb96b8ddf: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260620 09:30:01.727757 2090 heartbeater.cc:503] Master 127.16.43.190:36081 was elected leader, sending a full tablet report...
W20260620 09:30:01.759649 1960 tablet.cc:2369] T 364f6dacb9fa43b8a96aea8a54e4a2c4 P a8116ad43adb41e298726dfee1f24a8a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 09:30:01.807279 2109 raft_consensus.cc:482] T 364f6dacb9fa43b8a96aea8a54e4a2c4 P a8116ad43adb41e298726dfee1f24a8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:30:01.807416 2109 raft_consensus.cc:504] T 364f6dacb9fa43b8a96aea8a54e4a2c4 P a8116ad43adb41e298726dfee1f24a8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d15d5c052284c499e2fb50845fc1926" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 42071 } } peers { permanent_uuid: "a8116ad43adb41e298726dfee1f24a8a" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 46699 } } peers { permanent_uuid: "b62ec926ff704888a2b357bbb96b8ddf" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 46185 } }
I20260620 09:30:01.807832 2109 leader_election.cc:288] T 364f6dacb9fa43b8a96aea8a54e4a2c4 P a8116ad43adb41e298726dfee1f24a8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d15d5c052284c499e2fb50845fc1926 (127.16.43.129:42071), b62ec926ff704888a2b357bbb96b8ddf (127.16.43.131:46185)
I20260620 09:30:01.811779 2102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:01.808129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.130:57135 (local address 127.16.43.131:46185)
0620 09:30:01.808202 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:01.808204 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:01.808216 (+ 12us) server_negotiation.cc:407] Connection header received
0620 09:30:01.808284 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:01.808285 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:01.808295 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:01.808406 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:30:01.808602 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:01.809371 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:01.810778 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:01.811377 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:01.811457 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:01.811465 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:01.811660 (+ 195us) server_negotiation.cc:299] Negotiation successful
0620 09:30:01.811709 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 09:30:01.811785 1961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:01.808047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.131:46185 (local address 127.16.43.130:57135)
0620 09:30:01.808116 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0620 09:30:01.808128 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 09:30:01.808175 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:30:01.808440 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:30:01.808441 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:30:01.808443 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:30:01.808520 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:30:01.808523 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:30:01.809458 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:30:01.809459 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:30:01.810686 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:30:01.810689 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:30:01.811476 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:30:01.811477 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:30:01.811580 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:01.811587 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 09:30:01.811646 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 09:30:01.811694 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49,"mutex_wait_us":28}
I20260620 09:30:01.811820 2100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:01.808112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.130:34661 (local address 127.16.43.129:42071)
0620 09:30:01.808202 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:01.808204 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:01.808447 (+ 243us) server_negotiation.cc:407] Connection header received
0620 09:30:01.808578 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:01.808579 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:01.808593 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:01.808721 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:30:01.808875 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:01.809539 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:01.810843 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:01.811377 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:01.811457 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:01.811465 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:01.811712 (+ 247us) server_negotiation.cc:299] Negotiation successful
0620 09:30:01.811751 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 09:30:01.811796 2113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:01.808047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.129:42071 (local address 127.16.43.130:34661)
0620 09:30:01.808365 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0620 09:30:01.808381 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 09:30:01.808473 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:30:01.808741 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:30:01.808742 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:30:01.808744 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:30:01.808798 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:30:01.808799 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:30:01.809610 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:30:01.809610 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:30:01.810761 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:30:01.810763 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:30:01.811476 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:30:01.811477 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:30:01.811580 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:01.811587 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 09:30:01.811701 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 09:30:01.811729 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"mutex_wait_us":122,"thread_start_us":148,"threads_started":1}
I20260620 09:30:01.812304 1782 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "364f6dacb9fa43b8a96aea8a54e4a2c4" candidate_uuid: "a8116ad43adb41e298726dfee1f24a8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d15d5c052284c499e2fb50845fc1926" is_pre_election: true
I20260620 09:30:01.812330 2044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "364f6dacb9fa43b8a96aea8a54e4a2c4" candidate_uuid: "a8116ad43adb41e298726dfee1f24a8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b62ec926ff704888a2b357bbb96b8ddf" is_pre_election: true
I20260620 09:30:01.812472 2044 raft_consensus.cc:2454] T 364f6dacb9fa43b8a96aea8a54e4a2c4 P b62ec926ff704888a2b357bbb96b8ddf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8116ad43adb41e298726dfee1f24a8a in term 0.
I20260620 09:30:01.812474 1782 raft_consensus.cc:2454] T 364f6dacb9fa43b8a96aea8a54e4a2c4 P 6d15d5c052284c499e2fb50845fc1926 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8116ad43adb41e298726dfee1f24a8a in term 0.
I20260620 09:30:01.812736 1846 leader_election.cc:302] T 364f6dacb9fa43b8a96aea8a54e4a2c4 P a8116ad43adb41e298726dfee1f24a8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d15d5c052284c499e2fb50845fc1926, a8116ad43adb41e298726dfee1f24a8a; no voters:
I20260620 09:30:01.812916 2109 raft_consensus.cc:2781] T 364f6dacb9fa43b8a96aea8a54e4a2c4 P a8116ad43adb41e298726dfee1f24a8a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 09:30:01.812968 2109 raft_consensus.cc:482] T 364f6dacb9fa43b8a96aea8a54e4a2c4 P a8116ad43adb41e298726dfee1f24a8a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 09:30:01.812983 2109 raft_consensus.cc:3037] T 364f6dacb9fa43b8a96aea8a54e4a2c4 P a8116ad43adb41e298726dfee1f24a8a [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:30:01.813668 2109 raft_consensus.cc:504] T 364f6dacb9fa43b8a96aea8a54e4a2c4 P a8116ad43adb41e298726dfee1f24a8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d15d5c052284c499e2fb50845fc1926" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 42071 } } peers { permanent_uuid: "a8116ad43adb41e298726dfee1f24a8a" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 46699 } } peers { permanent_uuid: "b62ec926ff704888a2b357bbb96b8ddf" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 46185 } }
I20260620 09:30:01.813813 2109 leader_election.cc:288] T 364f6dacb9fa43b8a96aea8a54e4a2c4 P a8116ad43adb41e298726dfee1f24a8a [CANDIDATE]: Term 1 election: Requested vote from peers 6d15d5c052284c499e2fb50845fc1926 (127.16.43.129:42071), b62ec926ff704888a2b357bbb96b8ddf (127.16.43.131:46185)
I20260620 09:30:01.813962 1782 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "364f6dacb9fa43b8a96aea8a54e4a2c4" candidate_uuid: "a8116ad43adb41e298726dfee1f24a8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d15d5c052284c499e2fb50845fc1926"
I20260620 09:30:01.813972 2044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "364f6dacb9fa43b8a96aea8a54e4a2c4" candidate_uuid: "a8116ad43adb41e298726dfee1f24a8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b62ec926ff704888a2b357bbb96b8ddf"
I20260620 09:30:01.814055 1782 raft_consensus.cc:3037] T 364f6dacb9fa43b8a96aea8a54e4a2c4 P 6d15d5c052284c499e2fb50845fc1926 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:30:01.814050 2044 raft_consensus.cc:3037] T 364f6dacb9fa43b8a96aea8a54e4a2c4 P b62ec926ff704888a2b357bbb96b8ddf [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:30:01.814786 2044 raft_consensus.cc:2454] T 364f6dacb9fa43b8a96aea8a54e4a2c4 P b62ec926ff704888a2b357bbb96b8ddf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8116ad43adb41e298726dfee1f24a8a in term 1.
I20260620 09:30:01.814906 1782 raft_consensus.cc:2454] T 364f6dacb9fa43b8a96aea8a54e4a2c4 P 6d15d5c052284c499e2fb50845fc1926 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8116ad43adb41e298726dfee1f24a8a in term 1.
I20260620 09:30:01.814961 1849 leader_election.cc:302] T 364f6dacb9fa43b8a96aea8a54e4a2c4 P a8116ad43adb41e298726dfee1f24a8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a8116ad43adb41e298726dfee1f24a8a, b62ec926ff704888a2b357bbb96b8ddf; no voters:
I20260620 09:30:01.815091 2109 raft_consensus.cc:2781] T 364f6dacb9fa43b8a96aea8a54e4a2c4 P a8116ad43adb41e298726dfee1f24a8a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:30:01.815274 2109 raft_consensus.cc:686] T 364f6dacb9fa43b8a96aea8a54e4a2c4 P a8116ad43adb41e298726dfee1f24a8a [term 1 LEADER]: Becoming Leader. State: Replica: a8116ad43adb41e298726dfee1f24a8a, State: Running, Role: LEADER
I20260620 09:30:01.815377 2109 consensus_queue.cc:237] T 364f6dacb9fa43b8a96aea8a54e4a2c4 P a8116ad43adb41e298726dfee1f24a8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d15d5c052284c499e2fb50845fc1926" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 42071 } } peers { permanent_uuid: "a8116ad43adb41e298726dfee1f24a8a" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 46699 } } peers { permanent_uuid: "b62ec926ff704888a2b357bbb96b8ddf" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 46185 } }
I20260620 09:30:01.816450 1639 catalog_manager.cc:5495] T 364f6dacb9fa43b8a96aea8a54e4a2c4 P a8116ad43adb41e298726dfee1f24a8a reported cstate change: term changed from 0 to 1, leader changed from <none> to a8116ad43adb41e298726dfee1f24a8a (127.16.43.130). New cstate: current_term: 1 leader_uuid: "a8116ad43adb41e298726dfee1f24a8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d15d5c052284c499e2fb50845fc1926" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 42071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a8116ad43adb41e298726dfee1f24a8a" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 46699 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b62ec926ff704888a2b357bbb96b8ddf" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 46185 } health_report { overall_health: UNKNOWN } } }
I20260620 09:30:01.830017 2103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:01.827435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49682 (local address 127.16.43.130:46699)
0620 09:30:01.827505 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:01.827507 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:01.827642 (+ 135us) server_negotiation.cc:407] Connection header received
0620 09:30:01.827691 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:01.827691 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:01.827713 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:01.827794 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:30:01.827913 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:01.828414 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:01.829204 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:01.829414 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:01.829499 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:01.829642 (+ 143us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:30:01.829753 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:30:01.829842 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:01.829883 (+ 41us) server_negotiation.cc:299] Negotiation successful
0620 09:30:01.829907 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 09:30:01.832151 1782 raft_consensus.cc:1261] T 364f6dacb9fa43b8a96aea8a54e4a2c4 P 6d15d5c052284c499e2fb50845fc1926 [term 1 FOLLOWER]: Refusing update from remote peer a8116ad43adb41e298726dfee1f24a8a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:30:01.832163 2044 raft_consensus.cc:1261] T 364f6dacb9fa43b8a96aea8a54e4a2c4 P b62ec926ff704888a2b357bbb96b8ddf [term 1 FOLLOWER]: Refusing update from remote peer a8116ad43adb41e298726dfee1f24a8a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:30:01.832551 2109 consensus_queue.cc:1035] T 364f6dacb9fa43b8a96aea8a54e4a2c4 P a8116ad43adb41e298726dfee1f24a8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d15d5c052284c499e2fb50845fc1926" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 42071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:30:01.832702 2114 consensus_queue.cc:1035] T 364f6dacb9fa43b8a96aea8a54e4a2c4 P a8116ad43adb41e298726dfee1f24a8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b62ec926ff704888a2b357bbb96b8ddf" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:30:01.836275 2118 mvcc.cc:205] Tried to move back new op lower bound from 7298858196301008896 to 7298858196236603392. Current Snapshot: MvccSnapshot[applied={T|T < 7298858196301008896 or (T in {7298858196301008896})}]
I20260620 09:30:01.837378 1638 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44256:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:30:01.837491 1638 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44256:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:30:01.838307 1638 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a8b6d85b5b834a2997399e19bc49f4be: Sending DeleteTablet for 3 replicas of tablet 364f6dacb9fa43b8a96aea8a54e4a2c4
I20260620 09:30:01.838781 1762 tablet_service.cc:1514] Processing DeleteTablet for tablet 364f6dacb9fa43b8a96aea8a54e4a2c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:30:01 UTC) from {username='slave'} at 127.0.0.1:58948
I20260620 09:30:01.838788 2024 tablet_service.cc:1514] Processing DeleteTablet for tablet 364f6dacb9fa43b8a96aea8a54e4a2c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:30:01 UTC) from {username='slave'} at 127.0.0.1:38888
I20260620 09:30:01.839000 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 1700
I20260620 09:30:01.839072 1893 tablet_service.cc:1514] Processing DeleteTablet for tablet 364f6dacb9fa43b8a96aea8a54e4a2c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:30:01 UTC) from {username='slave'} at 127.0.0.1:49676
I20260620 09:30:01.839931 2128 tablet_replica.cc:330] T 364f6dacb9fa43b8a96aea8a54e4a2c4 P a8116ad43adb41e298726dfee1f24a8a: stopping tablet replica
I20260620 09:30:01.840044 2128 raft_consensus.cc:2229] T 364f6dacb9fa43b8a96aea8a54e4a2c4 P a8116ad43adb41e298726dfee1f24a8a [term 1 LEADER]: Raft consensus shutting down.
I20260620 09:30:01.840150 2128 raft_consensus.cc:2258] T 364f6dacb9fa43b8a96aea8a54e4a2c4 P a8116ad43adb41e298726dfee1f24a8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:30:01.840575 2128 ts_tablet_manager.cc:1905] T 364f6dacb9fa43b8a96aea8a54e4a2c4 P a8116ad43adb41e298726dfee1f24a8a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:30:01.842716 2128 ts_tablet_manager.cc:1918] T 364f6dacb9fa43b8a96aea8a54e4a2c4 P a8116ad43adb41e298726dfee1f24a8a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:30:01.842801 2128 log.cc:1196] T 364f6dacb9fa43b8a96aea8a54e4a2c4 P a8116ad43adb41e298726dfee1f24a8a: Deleting WAL directory at /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781947562949448-16558-0/minicluster-data/ts-1/wal/wals/364f6dacb9fa43b8a96aea8a54e4a2c4
I20260620 09:30:01.843109 2128 ts_tablet_manager.cc:1939] T 364f6dacb9fa43b8a96aea8a54e4a2c4 P a8116ad43adb41e298726dfee1f24a8a: Deleting consensus metadata
I20260620 09:30:01.843645 2126 tablet_replica.cc:330] T 364f6dacb9fa43b8a96aea8a54e4a2c4 P b62ec926ff704888a2b357bbb96b8ddf: stopping tablet replica
I20260620 09:30:01.843670 1627 catalog_manager.cc:4841] TS a8116ad43adb41e298726dfee1f24a8a (127.16.43.130:46699): tablet 364f6dacb9fa43b8a96aea8a54e4a2c4 (table test-table [id=0d3460de014f44678bdd551f4d25bc8b]) successfully deleted
I20260620 09:30:01.843873 2126 raft_consensus.cc:2229] T 364f6dacb9fa43b8a96aea8a54e4a2c4 P b62ec926ff704888a2b357bbb96b8ddf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:30:01.844014 2126 raft_consensus.cc:2258] T 364f6dacb9fa43b8a96aea8a54e4a2c4 P b62ec926ff704888a2b357bbb96b8ddf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:30:01.844540 2126 ts_tablet_manager.cc:1905] T 364f6dacb9fa43b8a96aea8a54e4a2c4 P b62ec926ff704888a2b357bbb96b8ddf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:30:01.846210 2126 ts_tablet_manager.cc:1918] T 364f6dacb9fa43b8a96aea8a54e4a2c4 P b62ec926ff704888a2b357bbb96b8ddf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:30:01.846299 2126 log.cc:1196] T 364f6dacb9fa43b8a96aea8a54e4a2c4 P b62ec926ff704888a2b357bbb96b8ddf: Deleting WAL directory at /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781947562949448-16558-0/minicluster-data/ts-2/wal/wals/364f6dacb9fa43b8a96aea8a54e4a2c4
I20260620 09:30:01.846550 2126 ts_tablet_manager.cc:1939] T 364f6dacb9fa43b8a96aea8a54e4a2c4 P b62ec926ff704888a2b357bbb96b8ddf: Deleting consensus metadata
W20260620 09:30:01.846720 1624 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.43.129:42071 (error 108)
I20260620 09:30:01.846943 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 1831
I20260620 09:30:01.846994 1627 catalog_manager.cc:4841] TS b62ec926ff704888a2b357bbb96b8ddf (127.16.43.131:46185): tablet 364f6dacb9fa43b8a96aea8a54e4a2c4 (table test-table [id=0d3460de014f44678bdd551f4d25bc8b]) successfully deleted
I20260620 09:30:01.847410 2098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:01.847281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.129:42071 (local address 127.0.0.1:58964)
0620 09:30:01.847330 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0620 09:30:01.847369 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.129:42071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260620 09:30:01.847687 1624 catalog_manager.cc:4584] TS 6d15d5c052284c499e2fb50845fc1926 (127.16.43.129:42071): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 364f6dacb9fa43b8a96aea8a54e4a2c4: Network error: Client connection negotiation failed: client connection to 127.16.43.129:42071: connect: Connection refused (error 111)
I20260620 09:30:01.853103 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 1962
I20260620 09:30:01.857460 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 1609
2026-06-20T09:30:01Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (794 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781947562949448-16558-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:30:01 dist-test-slave-706q krb5kdc[2132](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:30:01 dist-test-slave-706q krb5kdc[2132](info): set up 2 sockets
Jun 20 09:30:01 dist-test-slave-706q krb5kdc[2132](info): commencing operation
krb5kdc: starting...
W20260620 09:30:02.916491 16558 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781947562949448-16558-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781947562949448-16558-0/krb5kdc/test-user.keytab.
Jun 20 09:30:03 dist-test-slave-706q krb5kdc[2132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947803, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:30:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:30:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.43.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.43.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.43.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.190.keytab.
Entry for principal kudu/127.16.43.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.190.keytab.
I20260620 09:30:03.042799 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781947562949448-16558-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781947562949448-16558-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781947562949448-16558-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.43.190:41733
--webserver_interface=127.16.43.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.190.keytab
--principal=kudu/127.16.43.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.43.148:34027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.43.190:41733
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781947562949448-16558-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781947562949448-16558-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781947562949448-16558-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:30:03.124338 2145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:30:03.124625 2145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:30:03.124675 2145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:30:03.126391 2145 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:30:03.126470 2145 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:30:03.126493 2145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:30:03.126514 2145 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:30:03.126542 2145 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:30:03.128168 2145 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:34027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781947562949448-16558-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781947562949448-16558-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.43.190:41733
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.190.keytab
--principal=kudu/127.16.43.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.43.190:41733
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.43.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2145
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781947562949448-16558-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:30:03.128479 2145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:30:03.128731 2145 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:30:03.133924 2150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:30:03.133966 2151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:30:03.133929 2153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:30:03.134075 2145 server_base.cc:1034] running on GCE node
I20260620 09:30:03.134491 2145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:30:03.134884 2145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:30:03.136094 2145 hybrid_clock.cc:648] HybridClock initialized: now 1781947803136084 us; error 48 us; skew 500 ppm
Jun 20 09:30:03 dist-test-slave-706q krb5kdc[2132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947803, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:30:03.138880 2145 init.cc:377] Logged in from keytab as kudu/127.16.43.190@KRBTEST.COM (short username kudu)
I20260620 09:30:03.139374 2145 webserver.cc:460] Webserver started at http://127.16.43.190:35209/ using document root <none> and password file <none>
I20260620 09:30:03.139676 2145 fs_manager.cc:359] Metadata directory not provided
I20260620 09:30:03.139753 2145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:30:03.139905 2145 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:30:03.140992 2145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781947562949448-16558-0/minicluster-data/master-0/data/instance:
uuid: "25ac048c215b4fd7ab30c13dfba79761"
format_stamp: "Formatted at 2026-06-20 09:30:03 on dist-test-slave-706q"
server_key: "05692954c8bc6f05b35eddc99ec79e64"
server_key_iv: "f862401e6a2d0e5ca6b57f0b0a497c87"
server_key_version: "encryptionkey@0"
I20260620 09:30:03.141397 2145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781947562949448-16558-0/minicluster-data/master-0/wal/instance:
uuid: "25ac048c215b4fd7ab30c13dfba79761"
format_stamp: "Formatted at 2026-06-20 09:30:03 on dist-test-slave-706q"
server_key: "05692954c8bc6f05b35eddc99ec79e64"
server_key_iv: "f862401e6a2d0e5ca6b57f0b0a497c87"
server_key_version: "encryptionkey@0"
I20260620 09:30:03.142858 2145 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 09:30:03.143924 2160 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:30:03.144146 2145 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:30:03.144260 2145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781947562949448-16558-0/minicluster-data/master-0/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781947562949448-16558-0/minicluster-data/master-0/wal
uuid: "25ac048c215b4fd7ab30c13dfba79761"
format_stamp: "Formatted at 2026-06-20 09:30:03 on dist-test-slave-706q"
server_key: "05692954c8bc6f05b35eddc99ec79e64"
server_key_iv: "f862401e6a2d0e5ca6b57f0b0a497c87"
server_key_version: "encryptionkey@0"
I20260620 09:30:03.144351 2145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781947562949448-16558-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781947562949448-16558-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781947562949448-16558-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:30:03.150923 2145 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:30:03.153291 2145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:30:03.153476 2145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:30:03.157527 2145 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.190:41733
I20260620 09:30:03.157586 2212 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.190:41733 every 8 connection(s)
I20260620 09:30:03.157963 2145 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
I20260620 09:30:03.158895 2213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:30:03.161279 2213 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25ac048c215b4fd7ab30c13dfba79761: Bootstrap starting.
I20260620 09:30:03.161911 2213 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 25ac048c215b4fd7ab30c13dfba79761: Neither blocks nor log segments found. Creating new log.
I20260620 09:30:03.162210 2213 log.cc:824] T 00000000000000000000000000000000 P 25ac048c215b4fd7ab30c13dfba79761: Log is configured to *not* fsync() on all Append() calls
I20260620 09:30:03.163018 2213 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25ac048c215b4fd7ab30c13dfba79761: No bootstrap required, opened a new log
I20260620 09:30:03.164825 2213 raft_consensus.cc:348] T 00000000000000000000000000000000 P 25ac048c215b4fd7ab30c13dfba79761 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25ac048c215b4fd7ab30c13dfba79761" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 41733 } }
I20260620 09:30:03.164965 2213 raft_consensus.cc:374] T 00000000000000000000000000000000 P 25ac048c215b4fd7ab30c13dfba79761 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:30:03.165063 2213 raft_consensus.cc:729] T 00000000000000000000000000000000 P 25ac048c215b4fd7ab30c13dfba79761 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25ac048c215b4fd7ab30c13dfba79761, State: Initialized, Role: FOLLOWER
I20260620 09:30:03.165251 2213 consensus_queue.cc:260] T 00000000000000000000000000000000 P 25ac048c215b4fd7ab30c13dfba79761 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25ac048c215b4fd7ab30c13dfba79761" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 41733 } }
I20260620 09:30:03.165315 2213 raft_consensus.cc:388] T 00000000000000000000000000000000 P 25ac048c215b4fd7ab30c13dfba79761 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:30:03.165369 2213 raft_consensus.cc:482] T 00000000000000000000000000000000 P 25ac048c215b4fd7ab30c13dfba79761 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:30:03.165421 2213 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 25ac048c215b4fd7ab30c13dfba79761 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:30:03.165982 2213 raft_consensus.cc:504] T 00000000000000000000000000000000 P 25ac048c215b4fd7ab30c13dfba79761 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25ac048c215b4fd7ab30c13dfba79761" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 41733 } }
I20260620 09:30:03.166069 2213 leader_election.cc:302] T 00000000000000000000000000000000 P 25ac048c215b4fd7ab30c13dfba79761 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 25ac048c215b4fd7ab30c13dfba79761; no voters:
I20260620 09:30:03.166306 2213 leader_election.cc:288] T 00000000000000000000000000000000 P 25ac048c215b4fd7ab30c13dfba79761 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:30:03.166376 2216 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 25ac048c215b4fd7ab30c13dfba79761 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:30:03.166551 2216 raft_consensus.cc:686] T 00000000000000000000000000000000 P 25ac048c215b4fd7ab30c13dfba79761 [term 1 LEADER]: Becoming Leader. State: Replica: 25ac048c215b4fd7ab30c13dfba79761, State: Running, Role: LEADER
I20260620 09:30:03.166594 2213 sys_catalog.cc:564] T 00000000000000000000000000000000 P 25ac048c215b4fd7ab30c13dfba79761 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:30:03.166648 2216 consensus_queue.cc:237] T 00000000000000000000000000000000 P 25ac048c215b4fd7ab30c13dfba79761 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25ac048c215b4fd7ab30c13dfba79761" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 41733 } }
I20260620 09:30:03.167040 2218 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25ac048c215b4fd7ab30c13dfba79761 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 25ac048c215b4fd7ab30c13dfba79761. Latest consensus state: current_term: 1 leader_uuid: "25ac048c215b4fd7ab30c13dfba79761" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25ac048c215b4fd7ab30c13dfba79761" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 41733 } } }
I20260620 09:30:03.167127 2218 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25ac048c215b4fd7ab30c13dfba79761 [sys.catalog]: This master's current role is: LEADER
I20260620 09:30:03.167256 2217 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25ac048c215b4fd7ab30c13dfba79761 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "25ac048c215b4fd7ab30c13dfba79761" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25ac048c215b4fd7ab30c13dfba79761" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 41733 } } }
I20260620 09:30:03.167356 2224 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:30:03.167791 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 2145
I20260620 09:30:03.167356 2217 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25ac048c215b4fd7ab30c13dfba79761 [sys.catalog]: This master's current role is: LEADER
I20260620 09:30:03.167925 16558 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781947562949448-16558-0/minicluster-data/master-0/wal/instance
I20260620 09:30:03.168155 16558 external_mini_cluster.cc:1428] Setting key 2f43037ee296452f9974f7e3b4edb44e
I20260620 09:30:03.168208 2224 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:30:03.169816 2224 catalog_manager.cc:1347] Generated new cluster ID: cd7eb2750d884765ab09dfbc59f04e98
I20260620 09:30:03.169865 2224 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 09:30:03 dist-test-slave-706q krb5kdc[2132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947803, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.43.190@KRBTEST.COM
I20260620 09:30:03.180781 2235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:03.168606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44150 (local address 127.16.43.190:41733)
0620 09:30:03.169534 (+ 928us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:03.169544 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:03.169573 (+ 29us) server_negotiation.cc:407] Connection header received
0620 09:30:03.169698 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:03.169705 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:03.169758 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:03.170661 (+ 903us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:30:03.170753 (+ 92us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:03.172183 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:03.173250 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:03.173936 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:03.174050 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:03.176991 (+ 2941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:30:03.176996 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:30:03.176997 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:30:03.177010 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:30:03.179039 (+ 2029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:30:03.179469 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:30:03.179472 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:30:03.179472 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:30:03.179621 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:30:03.179834 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:30:03.179834 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:30:03.179835 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:30:03.180143 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:30:03.180284 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:03.180411 (+ 127us) server_negotiation.cc:299] Negotiation successful
0620 09:30:03.180544 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":875,"thread_start_us":82,"threads_started":1}
I20260620 09:30:03.184389 2224 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:30:03.184996 2224 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:30:03.189707 2224 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 25ac048c215b4fd7ab30c13dfba79761: Generated new TSK 0
I20260620 09:30:03.189899 2224 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.43.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.43.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.43.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.129.keytab.
Entry for principal kudu/127.16.43.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.129.keytab.
I20260620 09:30:03.219827 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781947562949448-16558-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781947562949448-16558-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781947562949448-16558-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781947562949448-16558-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.43.129:0
--local_ip_for_outbound_sockets=127.16.43.129
--webserver_interface=127.16.43.129
--webserver_port=0
--tserver_master_addrs=127.16.43.190:41733
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.129.keytab
--principal=kudu/127.16.43.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.43.148:34027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781947562949448-16558-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781947562949448-16558-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781947562949448-16558-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:30:03.303516 2239 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:30:03.303771 2239 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:30:03.303795 2239 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:30:03.305070 2239 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:30:03.305114 2239 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:30:03.305142 2239 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.43.129
I20260620 09:30:03.306604 2239 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:34027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781947562949448-16558-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781947562949448-16558-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.129.keytab
--principal=kudu/127.16.43.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.43.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781947562949448-16558-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.43.129
--webserver_port=0
--tserver_master_addrs=127.16.43.190:41733
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2239
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.43.129
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781947562949448-16558-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:30:03.306855 2239 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:30:03.307109 2239 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:30:03.312439 2244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:30:03.312465 2245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:30:03.312422 2247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:30:03.312695 2239 server_base.cc:1034] running on GCE node
I20260620 09:30:03.312938 2239 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:30:03.313212 2239 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:30:03.314397 2239 hybrid_clock.cc:648] HybridClock initialized: now 1781947803314384 us; error 27 us; skew 500 ppm
Jun 20 09:30:03 dist-test-slave-706q krb5kdc[2132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947803, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:30:03.317006 2239 init.cc:377] Logged in from keytab as kudu/127.16.43.129@KRBTEST.COM (short username kudu)
I20260620 09:30:03.317464 2239 webserver.cc:460] Webserver started at http://127.16.43.129:35343/ using document root <none> and password file <none>
I20260620 09:30:03.317723 2239 fs_manager.cc:359] Metadata directory not provided
I20260620 09:30:03.317793 2239 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:30:03.317945 2239 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:30:03.319010 2239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781947562949448-16558-0/minicluster-data/ts-0/data/instance:
uuid: "50d4531269a9423990ec81398b999df9"
format_stamp: "Formatted at 2026-06-20 09:30:03 on dist-test-slave-706q"
server_key: "ca29b5abd55914bf4b629dfa3d82f817"
server_key_iv: "a9a1111fa4b489926f35f7273f1e906f"
server_key_version: "encryptionkey@0"
I20260620 09:30:03.319375 2239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781947562949448-16558-0/minicluster-data/ts-0/wal/instance:
uuid: "50d4531269a9423990ec81398b999df9"
format_stamp: "Formatted at 2026-06-20 09:30:03 on dist-test-slave-706q"
server_key: "ca29b5abd55914bf4b629dfa3d82f817"
server_key_iv: "a9a1111fa4b489926f35f7273f1e906f"
server_key_version: "encryptionkey@0"
I20260620 09:30:03.320776 2239 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 09:30:03.321584 2254 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:30:03.321786 2239 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 09:30:03.321878 2239 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781947562949448-16558-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781947562949448-16558-0/minicluster-data/ts-0/wal
uuid: "50d4531269a9423990ec81398b999df9"
format_stamp: "Formatted at 2026-06-20 09:30:03 on dist-test-slave-706q"
server_key: "ca29b5abd55914bf4b629dfa3d82f817"
server_key_iv: "a9a1111fa4b489926f35f7273f1e906f"
server_key_version: "encryptionkey@0"
I20260620 09:30:03.321969 2239 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781947562949448-16558-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781947562949448-16558-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781947562949448-16558-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:30:03.356420 2239 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:30:03.358397 2239 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:30:03.358573 2239 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:30:03.358840 2239 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:30:03.359171 2239 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:30:03.359225 2239 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:30:03.359267 2239 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:30:03.359309 2239 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:30:03.365221 2239 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.129:46619
I20260620 09:30:03.365314 2367 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.129:46619 every 8 connection(s)
I20260620 09:30:03.365629 2239 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781947562949448-16558-0/minicluster-data/ts-0/data/info.pb
Jun 20 09:30:03 dist-test-slave-706q krb5kdc[2132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947803, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.129@KRBTEST.COM for kudu/127.16.43.190@KRBTEST.COM
I20260620 09:30:03.374991 2235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:03.366448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.129:59293 (local address 127.16.43.190:41733)
0620 09:30:03.366506 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:03.366508 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:03.366685 (+ 177us) server_negotiation.cc:407] Connection header received
0620 09:30:03.367291 (+ 606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:03.367292 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:03.367302 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:03.367387 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:30:03.368521 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:03.369082 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:03.370048 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:03.370256 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:03.370323 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:03.372409 (+ 2086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:30:03.372410 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:30:03.372410 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:30:03.372417 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:30:03.373837 (+ 1420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:30:03.374211 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:30:03.374212 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:30:03.374212 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:30:03.374260 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:30:03.374463 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:30:03.374463 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:30:03.374463 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:30:03.374567 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:30:03.374618 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:03.374852 (+ 234us) server_negotiation.cc:299] Negotiation successful
0620 09:30:03.374924 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 09:30:03.375142 2370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:03.366402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:41733 (local address 127.16.43.129:59293)
0620 09:30:03.366558 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 09:30:03.366599 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 09:30:03.367190 (+ 591us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:30:03.367444 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:30:03.367447 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:30:03.368143 (+ 696us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:30:03.368402 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:30:03.368407 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:30:03.369178 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:30:03.369179 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:30:03.369953 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:30:03.369955 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:30:03.370345 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:30:03.370345 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:30:03.370444 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:03.370903 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:30:03.370911 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:30:03.372321 (+ 1410us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:30:03.373916 (+ 1595us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:30:03.373922 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:30:03.373924 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:30:03.374135 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:30:03.374317 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:30:03.374317 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:30:03.374317 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:30:03.374410 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:30:03.374621 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:30:03.374625 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:30:03.374739 (+ 114us) client_negotiation.cc:769] Sending connection context
0620 09:30:03.374821 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 09:30:03.374937 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":89,"threads_started":1}
I20260620 09:30:03.375311 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 2239
I20260620 09:30:03.375423 16558 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781947562949448-16558-0/minicluster-data/ts-0/wal/instance
I20260620 09:30:03.375686 16558 external_mini_cluster.cc:1428] Setting key e0039f81ff733e956148b7d017a8d23d
I20260620 09:30:03.375674 2368 heartbeater.cc:347] Connected to a master server at 127.16.43.190:41733
I20260620 09:30:03.375809 2368 heartbeater.cc:464] Registering TS with master...
I20260620 09:30:03.376020 2368 heartbeater.cc:511] Master 127.16.43.190:41733 requested a full tablet report, sending...
I20260620 09:30:03.376715 2177 ts_manager.cc:194] Registered new tserver with Master: 50d4531269a9423990ec81398b999df9 (127.16.43.129:46619)
I20260620 09:30:03.377635 2177 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.43.129@KRBTEST.COM'} at 127.16.43.129:59293
WARNING: no policy specified for kudu/127.16.43.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.43.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.43.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.130.keytab.
Entry for principal kudu/127.16.43.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.130.keytab.
I20260620 09:30:03.403264 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781947562949448-16558-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781947562949448-16558-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781947562949448-16558-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781947562949448-16558-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.43.130:0
--local_ip_for_outbound_sockets=127.16.43.130
--webserver_interface=127.16.43.130
--webserver_port=0
--tserver_master_addrs=127.16.43.190:41733
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.130.keytab
--principal=kudu/127.16.43.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.43.148:34027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781947562949448-16558-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781947562949448-16558-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781947562949448-16558-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:30:03.486110 2373 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:30:03.486375 2373 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:30:03.486418 2373 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:30:03.487715 2373 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:30:03.487780 2373 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:30:03.487828 2373 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.43.130
I20260620 09:30:03.489347 2373 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:34027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781947562949448-16558-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781947562949448-16558-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.130.keytab
--principal=kudu/127.16.43.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.43.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781947562949448-16558-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.43.130
--webserver_port=0
--tserver_master_addrs=127.16.43.190:41733
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2373
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.43.130
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781947562949448-16558-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:30:03.489640 2373 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:30:03.489885 2373 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:30:03.495085 2378 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:30:03.495077 2379 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:30:03.495076 2381 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:30:03.495095 2373 server_base.cc:1034] running on GCE node
I20260620 09:30:03.495498 2373 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:30:03.495885 2373 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:30:03.497047 2373 hybrid_clock.cc:648] HybridClock initialized: now 1781947803497034 us; error 28 us; skew 500 ppm
Jun 20 09:30:03 dist-test-slave-706q krb5kdc[2132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947803, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:30:03.499784 2373 init.cc:377] Logged in from keytab as kudu/127.16.43.130@KRBTEST.COM (short username kudu)
I20260620 09:30:03.500255 2373 webserver.cc:460] Webserver started at http://127.16.43.130:43803/ using document root <none> and password file <none>
I20260620 09:30:03.500448 2373 fs_manager.cc:359] Metadata directory not provided
I20260620 09:30:03.500488 2373 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:30:03.500577 2373 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:30:03.501596 2373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781947562949448-16558-0/minicluster-data/ts-1/data/instance:
uuid: "af9718b2daad486b885084ffb2e5346d"
format_stamp: "Formatted at 2026-06-20 09:30:03 on dist-test-slave-706q"
server_key: "294bf87f58974b69b4222f5b0ce6cd6f"
server_key_iv: "a6ab8ea24a5cc500294f136504c7657f"
server_key_version: "encryptionkey@0"
I20260620 09:30:03.502053 2373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781947562949448-16558-0/minicluster-data/ts-1/wal/instance:
uuid: "af9718b2daad486b885084ffb2e5346d"
format_stamp: "Formatted at 2026-06-20 09:30:03 on dist-test-slave-706q"
server_key: "294bf87f58974b69b4222f5b0ce6cd6f"
server_key_iv: "a6ab8ea24a5cc500294f136504c7657f"
server_key_version: "encryptionkey@0"
I20260620 09:30:03.503362 2373 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260620 09:30:03.504351 2388 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:30:03.504599 2373 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:30:03.504704 2373 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781947562949448-16558-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781947562949448-16558-0/minicluster-data/ts-1/wal
uuid: "af9718b2daad486b885084ffb2e5346d"
format_stamp: "Formatted at 2026-06-20 09:30:03 on dist-test-slave-706q"
server_key: "294bf87f58974b69b4222f5b0ce6cd6f"
server_key_iv: "a6ab8ea24a5cc500294f136504c7657f"
server_key_version: "encryptionkey@0"
I20260620 09:30:03.504787 2373 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781947562949448-16558-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781947562949448-16558-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781947562949448-16558-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:30:03.524701 2373 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:30:03.527107 2373 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:30:03.527280 2373 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:30:03.527577 2373 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:30:03.527917 2373 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:30:03.527956 2373 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:30:03.527980 2373 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:30:03.528026 2373 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:30:03.534129 2373 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.130:34711
I20260620 09:30:03.534199 2501 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.130:34711 every 8 connection(s)
I20260620 09:30:03.534509 2373 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781947562949448-16558-0/minicluster-data/ts-1/data/info.pb
I20260620 09:30:03.538820 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 2373
I20260620 09:30:03.538954 16558 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781947562949448-16558-0/minicluster-data/ts-1/wal/instance
I20260620 09:30:03.539160 16558 external_mini_cluster.cc:1428] Setting key 0361d25572bd61439e08057126cce745
Jun 20 09:30:03 dist-test-slave-706q krb5kdc[2132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947803, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.130@KRBTEST.COM for kudu/127.16.43.190@KRBTEST.COM
I20260620 09:30:03.545214 2235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:03.535433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.130:44111 (local address 127.16.43.190:41733)
0620 09:30:03.535475 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:03.535477 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:03.535725 (+ 248us) server_negotiation.cc:407] Connection header received
0620 09:30:03.536201 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:03.536202 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:03.536213 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:03.536281 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:30:03.537216 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:03.537713 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:03.538919 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:03.539169 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:03.539271 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:03.542024 (+ 2753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:30:03.542026 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:30:03.542026 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:30:03.542033 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:30:03.543681 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:30:03.544180 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:30:03.544181 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:30:03.544181 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:30:03.544230 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:30:03.544523 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:30:03.544523 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:30:03.544523 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:30:03.544630 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:30:03.544711 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:03.545077 (+ 366us) server_negotiation.cc:299] Negotiation successful
0620 09:30:03.545146 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 09:30:03.545552 2504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:03.535424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:41733 (local address 127.16.43.130:44111)
0620 09:30:03.535599 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0620 09:30:03.535635 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 09:30:03.536105 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:30:03.536339 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:30:03.536342 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:30:03.536933 (+ 591us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:30:03.537132 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:30:03.537136 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:30:03.537818 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:30:03.537819 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:30:03.538786 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:30:03.538790 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:30:03.539308 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:30:03.539310 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:30:03.539446 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:03.540137 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:30:03.540162 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:30:03.541918 (+ 1756us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:30:03.543769 (+ 1851us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:30:03.543777 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:30:03.543780 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:30:03.544106 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:30:03.544297 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:30:03.544298 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:30:03.544298 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:30:03.544452 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:30:03.544727 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:30:03.544740 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:30:03.544928 (+ 188us) client_negotiation.cc:769] Sending connection context
0620 09:30:03.545062 (+ 134us) client_negotiation.cc:240] Negotiation successful
0620 09:30:03.545257 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":126,"threads_started":1}
I20260620 09:30:03.546192 2502 heartbeater.cc:347] Connected to a master server at 127.16.43.190:41733
I20260620 09:30:03.546284 2502 heartbeater.cc:464] Registering TS with master...
I20260620 09:30:03.546489 2502 heartbeater.cc:511] Master 127.16.43.190:41733 requested a full tablet report, sending...
I20260620 09:30:03.546916 2177 ts_manager.cc:194] Registered new tserver with Master: af9718b2daad486b885084ffb2e5346d (127.16.43.130:34711)
I20260620 09:30:03.547489 2177 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.43.130@KRBTEST.COM'} at 127.16.43.130:44111
WARNING: no policy specified for kudu/127.16.43.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.43.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.43.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.131.keytab.
Entry for principal kudu/127.16.43.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.131.keytab.
I20260620 09:30:03.567018 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781947562949448-16558-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781947562949448-16558-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781947562949448-16558-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781947562949448-16558-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.43.131:0
--local_ip_for_outbound_sockets=127.16.43.131
--webserver_interface=127.16.43.131
--webserver_port=0
--tserver_master_addrs=127.16.43.190:41733
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.131.keytab
--principal=kudu/127.16.43.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.43.148:34027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781947562949448-16558-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781947562949448-16558-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781947562949448-16558-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:30:03.648728 2507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:30:03.648949 2507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:30:03.648978 2507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:30:03.650240 2507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:30:03.650285 2507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:30:03.650313 2507 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.43.131
I20260620 09:30:03.651966 2507 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:34027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781947562949448-16558-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781947562949448-16558-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.131.keytab
--principal=kudu/127.16.43.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.43.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781947562949448-16558-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.43.131
--webserver_port=0
--tserver_master_addrs=127.16.43.190:41733
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2507
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.43.131
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781947562949448-16558-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:30:03.652225 2507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:30:03.652480 2507 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 09:30:03.657903 2507 server_base.cc:1034] running on GCE node
W20260620 09:30:03.657891 2513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:30:03.657876 2512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:30:03.657886 2515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:30:03.658305 2507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:30:03.658597 2507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:30:03.659789 2507 hybrid_clock.cc:648] HybridClock initialized: now 1781947803659757 us; error 42 us; skew 500 ppm
Jun 20 09:30:03 dist-test-slave-706q krb5kdc[2132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947803, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:30:03.662322 2507 init.cc:377] Logged in from keytab as kudu/127.16.43.131@KRBTEST.COM (short username kudu)
I20260620 09:30:03.662869 2507 webserver.cc:460] Webserver started at http://127.16.43.131:40133/ using document root <none> and password file <none>
I20260620 09:30:03.663111 2507 fs_manager.cc:359] Metadata directory not provided
I20260620 09:30:03.663182 2507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:30:03.663316 2507 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:30:03.664458 2507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781947562949448-16558-0/minicluster-data/ts-2/data/instance:
uuid: "161eb58fed224099b32195c6e8cfcbc7"
format_stamp: "Formatted at 2026-06-20 09:30:03 on dist-test-slave-706q"
server_key: "4f0786721f2de59bdbb62fee99e11046"
server_key_iv: "4097efc71a23b95405f11ece6229f1e4"
server_key_version: "encryptionkey@0"
I20260620 09:30:03.664842 2507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781947562949448-16558-0/minicluster-data/ts-2/wal/instance:
uuid: "161eb58fed224099b32195c6e8cfcbc7"
format_stamp: "Formatted at 2026-06-20 09:30:03 on dist-test-slave-706q"
server_key: "4f0786721f2de59bdbb62fee99e11046"
server_key_iv: "4097efc71a23b95405f11ece6229f1e4"
server_key_version: "encryptionkey@0"
I20260620 09:30:03.666188 2507 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 09:30:03.667014 2522 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:30:03.667189 2507 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 09:30:03.667290 2507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781947562949448-16558-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781947562949448-16558-0/minicluster-data/ts-2/wal
uuid: "161eb58fed224099b32195c6e8cfcbc7"
format_stamp: "Formatted at 2026-06-20 09:30:03 on dist-test-slave-706q"
server_key: "4f0786721f2de59bdbb62fee99e11046"
server_key_iv: "4097efc71a23b95405f11ece6229f1e4"
server_key_version: "encryptionkey@0"
I20260620 09:30:03.667368 2507 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781947562949448-16558-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781947562949448-16558-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781947562949448-16558-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:30:03.676221 2507 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:30:03.678436 2507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:30:03.678604 2507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:30:03.678845 2507 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:30:03.679177 2507 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:30:03.679232 2507 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:30:03.679275 2507 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:30:03.679302 2507 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:30:03.685294 2507 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.131:37177
I20260620 09:30:03.685372 2635 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.131:37177 every 8 connection(s)
I20260620 09:30:03.685719 2507 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781947562949448-16558-0/minicluster-data/ts-2/data/info.pb
Jun 20 09:30:03 dist-test-slave-706q krb5kdc[2132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947803, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.131@KRBTEST.COM for kudu/127.16.43.190@KRBTEST.COM
I20260620 09:30:03.692315 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 2507
I20260620 09:30:03.692428 16558 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781947562949448-16558-0/minicluster-data/ts-2/wal/instance
I20260620 09:30:03.692641 16558 external_mini_cluster.cc:1428] Setting key 652dac583507cfb1f19c05c4b3cb3a6c
I20260620 09:30:03.695643 2235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:03.686657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.131:44159 (local address 127.16.43.190:41733)
0620 09:30:03.686722 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:03.686724 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:03.686765 (+ 41us) server_negotiation.cc:407] Connection header received
0620 09:30:03.687303 (+ 538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:03.687304 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:03.687315 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:03.687383 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:30:03.688418 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:03.688968 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:03.690052 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:03.690244 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:03.690302 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:03.692699 (+ 2397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:30:03.692701 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:30:03.692701 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:30:03.692718 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:30:03.694160 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:30:03.694609 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:30:03.694610 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:30:03.694611 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:30:03.694674 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:30:03.694935 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:30:03.694936 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:30:03.694936 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:30:03.695056 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:30:03.695136 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:03.695425 (+ 289us) server_negotiation.cc:299] Negotiation successful
0620 09:30:03.695507 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 09:30:03.695897 2638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:03.686474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:41733 (local address 127.16.43.131:44159)
0620 09:30:03.686641 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 09:30:03.686684 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 09:30:03.687186 (+ 502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:30:03.687440 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:30:03.687444 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:30:03.688101 (+ 657us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:30:03.688328 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:30:03.688333 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:30:03.689054 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:30:03.689055 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:30:03.689977 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:30:03.689979 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:30:03.690368 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:30:03.690370 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:30:03.690528 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:03.691030 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:30:03.691045 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:30:03.692583 (+ 1538us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:30:03.694267 (+ 1684us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:30:03.694275 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:30:03.694278 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:30:03.694516 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:30:03.694743 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:30:03.694744 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:30:03.694744 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:30:03.694865 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:30:03.695126 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:30:03.695133 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:30:03.695287 (+ 154us) client_negotiation.cc:769] Sending connection context
0620 09:30:03.695402 (+ 115us) client_negotiation.cc:240] Negotiation successful
0620 09:30:03.695614 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":95,"threads_started":1}
I20260620 09:30:03.696431 2636 heartbeater.cc:347] Connected to a master server at 127.16.43.190:41733
I20260620 09:30:03.696540 2636 heartbeater.cc:464] Registering TS with master...
I20260620 09:30:03.696730 2636 heartbeater.cc:511] Master 127.16.43.190:41733 requested a full tablet report, sending...
I20260620 09:30:03.697119 2177 ts_manager.cc:194] Registered new tserver with Master: 161eb58fed224099b32195c6e8cfcbc7 (127.16.43.131:37177)
I20260620 09:30:03.697533 2177 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.43.131@KRBTEST.COM'} at 127.16.43.131:44159
I20260620 09:30:03.704496 16558 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 09:30:03.712759 2235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:03.705949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44164 (local address 127.16.43.190:41733)
0620 09:30:03.706032 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:03.706035 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:03.706126 (+ 91us) server_negotiation.cc:407] Connection header received
0620 09:30:03.706226 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:03.706227 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:03.706237 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:03.706309 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:30:03.706893 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:03.707384 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:03.708202 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:03.708408 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:03.708496 (+ 88us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:03.709901 (+ 1405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:30:03.709903 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:30:03.709903 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:30:03.709911 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:30:03.711334 (+ 1423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:30:03.711852 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:30:03.711853 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:30:03.711854 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:30:03.711917 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:30:03.712146 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:30:03.712147 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:30:03.712147 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:30:03.712305 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:30:03.712425 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:03.712561 (+ 136us) server_negotiation.cc:299] Negotiation successful
0620 09:30:03.712626 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 09:30:03.714663 2177 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44164:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 09:30:03.716447 2177 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 09:30:03.723371 2649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:03.718989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.131:37177 (local address 127.0.0.1:49852)
0620 09:30:03.719210 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0620 09:30:03.719219 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 09:30:03.719273 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:30:03.719878 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:30:03.719879 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:30:03.719881 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:30:03.719949 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:30:03.719952 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:30:03.720881 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:30:03.720882 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:30:03.722120 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:30:03.722123 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:30:03.723083 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:30:03.723084 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:30:03.723158 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:03.723163 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 09:30:03.723198 (+ 35us) client_negotiation.cc:240] Negotiation successful
0620 09:30:03.723275 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":116,"threads_started":1}
I20260620 09:30:03.723580 2647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:03.718977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.130:34711 (local address 127.0.0.1:43210)
0620 09:30:03.719165 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0620 09:30:03.719193 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0620 09:30:03.719273 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:30:03.719789 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:30:03.719793 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:30:03.719795 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:30:03.719910 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:30:03.719914 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:30:03.721291 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:30:03.721292 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:30:03.722387 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:30:03.722389 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:30:03.723322 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:30:03.723323 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:30:03.723391 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:03.723395 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 09:30:03.723452 (+ 57us) client_negotiation.cc:240] Negotiation successful
0620 09:30:03.723486 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"mutex_wait_us":30,"thread_start_us":90,"threads_started":1}
I20260620 09:30:03.723613 2648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:03.718977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.129:46619 (local address 127.0.0.1:39432)
0620 09:30:03.719152 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0620 09:30:03.719353 (+ 201us) client_negotiation.cc:174] Beginning negotiation
0620 09:30:03.719384 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:30:03.719790 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:30:03.719793 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:30:03.719797 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:30:03.719910 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:30:03.719913 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:30:03.721305 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:30:03.721306 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:30:03.722270 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:30:03.722273 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:30:03.723019 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:30:03.723020 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:30:03.723113 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:03.723129 (+ 16us) client_negotiation.cc:769] Sending connection context
0620 09:30:03.723324 (+ 195us) client_negotiation.cc:240] Negotiation successful
0620 09:30:03.723375 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"mutex_wait_us":28,"thread_start_us":61,"threads_started":1}
I20260620 09:30:03.723656 2650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:03.719178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43210 (local address 127.16.43.130:34711)
0620 09:30:03.719578 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:03.719582 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:03.719591 (+ 9us) server_negotiation.cc:407] Connection header received
0620 09:30:03.719628 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:03.719630 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:03.719646 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:03.719993 (+ 347us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:30:03.720062 (+ 69us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:03.721183 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:03.722475 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:03.723199 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:03.723280 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:03.723332 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:03.723488 (+ 156us) server_negotiation.cc:299] Negotiation successful
0620 09:30:03.723564 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":82,"threads_started":1}
I20260620 09:30:03.723925 2652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:03.719357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39432 (local address 127.16.43.129:46619)
0620 09:30:03.719568 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:03.719574 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:03.719585 (+ 11us) server_negotiation.cc:407] Connection header received
0620 09:30:03.719628 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:03.719630 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:03.719646 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:03.719984 (+ 338us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:30:03.720062 (+ 78us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:03.721174 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:03.722370 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:03.722927 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:03.723018 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:03.723091 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:03.723765 (+ 674us) server_negotiation.cc:299] Negotiation successful
0620 09:30:03.723845 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":86,"threads_started":1}
I20260620 09:30:03.724366 2436 tablet_service.cc:1467] Processing CreateTablet for tablet 964f3994481c4e9e9d56e8f17dbaa36f (DEFAULT_TABLE table=test-table [id=cb62ecb769754c6081a79f6d74e39aa2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:30:03.724512 2302 tablet_service.cc:1467] Processing CreateTablet for tablet 964f3994481c4e9e9d56e8f17dbaa36f (DEFAULT_TABLE table=test-table [id=cb62ecb769754c6081a79f6d74e39aa2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:30:03.724776 2436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 964f3994481c4e9e9d56e8f17dbaa36f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:30:03.724797 2302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 964f3994481c4e9e9d56e8f17dbaa36f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:30:03.727833 2651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:03.719260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49852 (local address 127.16.43.131:37177)
0620 09:30:03.719665 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:03.719669 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:03.719695 (+ 26us) server_negotiation.cc:407] Connection header received
0620 09:30:03.719753 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:03.719755 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:03.719771 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:03.720029 (+ 258us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:30:03.720093 (+ 64us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:03.720770 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:03.722207 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:03.722978 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:03.727610 (+ 4632us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:03.727694 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:03.727721 (+ 27us) server_negotiation.cc:299] Negotiation successful
0620 09:30:03.727765 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":81,"threads_started":1}
I20260620 09:30:03.728238 2654 tablet_bootstrap.cc:492] T 964f3994481c4e9e9d56e8f17dbaa36f P af9718b2daad486b885084ffb2e5346d: Bootstrap starting.
I20260620 09:30:03.728614 2653 tablet_bootstrap.cc:492] T 964f3994481c4e9e9d56e8f17dbaa36f P 50d4531269a9423990ec81398b999df9: Bootstrap starting.
I20260620 09:30:03.728538 2570 tablet_service.cc:1467] Processing CreateTablet for tablet 964f3994481c4e9e9d56e8f17dbaa36f (DEFAULT_TABLE table=test-table [id=cb62ecb769754c6081a79f6d74e39aa2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:30:03.728895 2570 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 964f3994481c4e9e9d56e8f17dbaa36f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:30:03.729084 2654 tablet_bootstrap.cc:654] T 964f3994481c4e9e9d56e8f17dbaa36f P af9718b2daad486b885084ffb2e5346d: Neither blocks nor log segments found. Creating new log.
I20260620 09:30:03.729306 2653 tablet_bootstrap.cc:654] T 964f3994481c4e9e9d56e8f17dbaa36f P 50d4531269a9423990ec81398b999df9: Neither blocks nor log segments found. Creating new log.
I20260620 09:30:03.729653 2654 log.cc:824] T 964f3994481c4e9e9d56e8f17dbaa36f P af9718b2daad486b885084ffb2e5346d: Log is configured to *not* fsync() on all Append() calls
I20260620 09:30:03.729812 2653 log.cc:824] T 964f3994481c4e9e9d56e8f17dbaa36f P 50d4531269a9423990ec81398b999df9: Log is configured to *not* fsync() on all Append() calls
I20260620 09:30:03.730553 2654 tablet_bootstrap.cc:492] T 964f3994481c4e9e9d56e8f17dbaa36f P af9718b2daad486b885084ffb2e5346d: No bootstrap required, opened a new log
I20260620 09:30:03.730633 2654 ts_tablet_manager.cc:1397] T 964f3994481c4e9e9d56e8f17dbaa36f P af9718b2daad486b885084ffb2e5346d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 09:30:03.730582 2653 tablet_bootstrap.cc:492] T 964f3994481c4e9e9d56e8f17dbaa36f P 50d4531269a9423990ec81398b999df9: No bootstrap required, opened a new log
I20260620 09:30:03.730718 2653 ts_tablet_manager.cc:1397] T 964f3994481c4e9e9d56e8f17dbaa36f P 50d4531269a9423990ec81398b999df9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 09:30:03.732136 2657 tablet_bootstrap.cc:492] T 964f3994481c4e9e9d56e8f17dbaa36f P 161eb58fed224099b32195c6e8cfcbc7: Bootstrap starting.
I20260620 09:30:03.732873 2657 tablet_bootstrap.cc:654] T 964f3994481c4e9e9d56e8f17dbaa36f P 161eb58fed224099b32195c6e8cfcbc7: Neither blocks nor log segments found. Creating new log.
I20260620 09:30:03.732748 2654 raft_consensus.cc:348] T 964f3994481c4e9e9d56e8f17dbaa36f P af9718b2daad486b885084ffb2e5346d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af9718b2daad486b885084ffb2e5346d" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 34711 } } peers { permanent_uuid: "50d4531269a9423990ec81398b999df9" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 46619 } } peers { permanent_uuid: "161eb58fed224099b32195c6e8cfcbc7" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 37177 } }
I20260620 09:30:03.732748 2653 raft_consensus.cc:348] T 964f3994481c4e9e9d56e8f17dbaa36f P 50d4531269a9423990ec81398b999df9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af9718b2daad486b885084ffb2e5346d" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 34711 } } peers { permanent_uuid: "50d4531269a9423990ec81398b999df9" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 46619 } } peers { permanent_uuid: "161eb58fed224099b32195c6e8cfcbc7" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 37177 } }
I20260620 09:30:03.732942 2653 raft_consensus.cc:374] T 964f3994481c4e9e9d56e8f17dbaa36f P 50d4531269a9423990ec81398b999df9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:30:03.732942 2654 raft_consensus.cc:374] T 964f3994481c4e9e9d56e8f17dbaa36f P af9718b2daad486b885084ffb2e5346d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:30:03.732988 2653 raft_consensus.cc:729] T 964f3994481c4e9e9d56e8f17dbaa36f P 50d4531269a9423990ec81398b999df9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50d4531269a9423990ec81398b999df9, State: Initialized, Role: FOLLOWER
I20260620 09:30:03.732988 2654 raft_consensus.cc:729] T 964f3994481c4e9e9d56e8f17dbaa36f P af9718b2daad486b885084ffb2e5346d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af9718b2daad486b885084ffb2e5346d, State: Initialized, Role: FOLLOWER
I20260620 09:30:03.733104 2654 consensus_queue.cc:260] T 964f3994481c4e9e9d56e8f17dbaa36f P af9718b2daad486b885084ffb2e5346d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af9718b2daad486b885084ffb2e5346d" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 34711 } } peers { permanent_uuid: "50d4531269a9423990ec81398b999df9" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 46619 } } peers { permanent_uuid: "161eb58fed224099b32195c6e8cfcbc7" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 37177 } }
I20260620 09:30:03.733104 2653 consensus_queue.cc:260] T 964f3994481c4e9e9d56e8f17dbaa36f P 50d4531269a9423990ec81398b999df9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af9718b2daad486b885084ffb2e5346d" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 34711 } } peers { permanent_uuid: "50d4531269a9423990ec81398b999df9" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 46619 } } peers { permanent_uuid: "161eb58fed224099b32195c6e8cfcbc7" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 37177 } }
I20260620 09:30:03.733383 2653 ts_tablet_manager.cc:1428] T 964f3994481c4e9e9d56e8f17dbaa36f P 50d4531269a9423990ec81398b999df9: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 09:30:03.733383 2654 ts_tablet_manager.cc:1428] T 964f3994481c4e9e9d56e8f17dbaa36f P af9718b2daad486b885084ffb2e5346d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 09:30:03.733484 2657 log.cc:824] T 964f3994481c4e9e9d56e8f17dbaa36f P 161eb58fed224099b32195c6e8cfcbc7: Log is configured to *not* fsync() on all Append() calls
I20260620 09:30:03.733517 2502 heartbeater.cc:503] Master 127.16.43.190:41733 was elected leader, sending a full tablet report...
I20260620 09:30:03.733554 2368 heartbeater.cc:503] Master 127.16.43.190:41733 was elected leader, sending a full tablet report...
I20260620 09:30:03.734326 2657 tablet_bootstrap.cc:492] T 964f3994481c4e9e9d56e8f17dbaa36f P 161eb58fed224099b32195c6e8cfcbc7: No bootstrap required, opened a new log
I20260620 09:30:03.734405 2657 ts_tablet_manager.cc:1397] T 964f3994481c4e9e9d56e8f17dbaa36f P 161eb58fed224099b32195c6e8cfcbc7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 09:30:03.736011 2657 raft_consensus.cc:348] T 964f3994481c4e9e9d56e8f17dbaa36f P 161eb58fed224099b32195c6e8cfcbc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af9718b2daad486b885084ffb2e5346d" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 34711 } } peers { permanent_uuid: "50d4531269a9423990ec81398b999df9" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 46619 } } peers { permanent_uuid: "161eb58fed224099b32195c6e8cfcbc7" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 37177 } }
I20260620 09:30:03.736177 2657 raft_consensus.cc:374] T 964f3994481c4e9e9d56e8f17dbaa36f P 161eb58fed224099b32195c6e8cfcbc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:30:03.736241 2657 raft_consensus.cc:729] T 964f3994481c4e9e9d56e8f17dbaa36f P 161eb58fed224099b32195c6e8cfcbc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 161eb58fed224099b32195c6e8cfcbc7, State: Initialized, Role: FOLLOWER
I20260620 09:30:03.736356 2657 consensus_queue.cc:260] T 964f3994481c4e9e9d56e8f17dbaa36f P 161eb58fed224099b32195c6e8cfcbc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af9718b2daad486b885084ffb2e5346d" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 34711 } } peers { permanent_uuid: "50d4531269a9423990ec81398b999df9" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 46619 } } peers { permanent_uuid: "161eb58fed224099b32195c6e8cfcbc7" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 37177 } }
I20260620 09:30:03.736634 2657 ts_tablet_manager.cc:1428] T 964f3994481c4e9e9d56e8f17dbaa36f P 161eb58fed224099b32195c6e8cfcbc7: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 09:30:03.736735 2636 heartbeater.cc:503] Master 127.16.43.190:41733 was elected leader, sending a full tablet report...
W20260620 09:30:03.785245 2503 tablet.cc:2369] T 964f3994481c4e9e9d56e8f17dbaa36f P af9718b2daad486b885084ffb2e5346d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 09:30:03.866401 2369 tablet.cc:2369] T 964f3994481c4e9e9d56e8f17dbaa36f P 50d4531269a9423990ec81398b999df9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 09:30:03.936352 2637 tablet.cc:2369] T 964f3994481c4e9e9d56e8f17dbaa36f P 161eb58fed224099b32195c6e8cfcbc7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 09:30:03.967303 2661 raft_consensus.cc:482] T 964f3994481c4e9e9d56e8f17dbaa36f P 161eb58fed224099b32195c6e8cfcbc7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:30:03.967497 2661 raft_consensus.cc:504] T 964f3994481c4e9e9d56e8f17dbaa36f P 161eb58fed224099b32195c6e8cfcbc7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af9718b2daad486b885084ffb2e5346d" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 34711 } } peers { permanent_uuid: "50d4531269a9423990ec81398b999df9" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 46619 } } peers { permanent_uuid: "161eb58fed224099b32195c6e8cfcbc7" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 37177 } }
I20260620 09:30:03.967955 2661 leader_election.cc:288] T 964f3994481c4e9e9d56e8f17dbaa36f P 161eb58fed224099b32195c6e8cfcbc7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af9718b2daad486b885084ffb2e5346d (127.16.43.130:34711), 50d4531269a9423990ec81398b999df9 (127.16.43.129:46619)
I20260620 09:30:03.971091 2650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:03.968304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.131:46887 (local address 127.16.43.130:34711)
0620 09:30:03.968354 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:03.968356 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:03.968365 (+ 9us) server_negotiation.cc:407] Connection header received
0620 09:30:03.968396 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:03.968397 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:03.968409 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:03.968463 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:30:03.968634 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:03.969112 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:03.970188 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:03.970712 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:03.970798 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:03.970808 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:03.970976 (+ 168us) server_negotiation.cc:299] Negotiation successful
0620 09:30:03.971017 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 09:30:03.971091 2638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:03.968171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.130:34711 (local address 127.16.43.131:46887)
0620 09:30:03.968212 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0620 09:30:03.968225 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 09:30:03.968312 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:30:03.968467 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:30:03.968468 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:30:03.968469 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:30:03.968551 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:30:03.968552 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:30:03.969208 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:30:03.969209 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:30:03.970054 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:30:03.970057 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:30:03.970797 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:30:03.970797 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:30:03.970884 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:03.970888 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 09:30:03.970966 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 09:30:03.971011 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18}
I20260620 09:30:03.971311 2652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:03.968364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.131:39579 (local address 127.16.43.129:46619)
0620 09:30:03.968416 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:03.968418 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:03.968587 (+ 169us) server_negotiation.cc:407] Connection header received
0620 09:30:03.968719 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:03.968720 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:03.968730 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:03.968814 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:30:03.968981 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:03.969466 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:03.970441 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:03.970908 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:03.970992 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:03.971002 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:03.971169 (+ 167us) server_negotiation.cc:299] Negotiation successful
0620 09:30:03.971223 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 09:30:03.971424 2662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:03.968285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.129:46619 (local address 127.16.43.131:39579)
0620 09:30:03.968499 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0620 09:30:03.968508 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 09:30:03.968602 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:30:03.968819 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:30:03.968820 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:30:03.968821 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:30:03.968896 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:30:03.968898 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:30:03.969581 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:30:03.969582 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:30:03.970348 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:30:03.970350 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:30:03.971004 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:30:03.971005 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:30:03.971085 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:03.971089 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 09:30:03.971288 (+ 199us) client_negotiation.cc:240] Negotiation successful
0620 09:30:03.971332 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":129,"threads_started":1}
I20260620 09:30:03.971760 2322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "964f3994481c4e9e9d56e8f17dbaa36f" candidate_uuid: "161eb58fed224099b32195c6e8cfcbc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50d4531269a9423990ec81398b999df9" is_pre_election: true
I20260620 09:30:03.971760 2456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "964f3994481c4e9e9d56e8f17dbaa36f" candidate_uuid: "161eb58fed224099b32195c6e8cfcbc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af9718b2daad486b885084ffb2e5346d" is_pre_election: true
I20260620 09:30:03.971926 2456 raft_consensus.cc:2454] T 964f3994481c4e9e9d56e8f17dbaa36f P af9718b2daad486b885084ffb2e5346d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 161eb58fed224099b32195c6e8cfcbc7 in term 0.
I20260620 09:30:03.971930 2322 raft_consensus.cc:2454] T 964f3994481c4e9e9d56e8f17dbaa36f P 50d4531269a9423990ec81398b999df9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 161eb58fed224099b32195c6e8cfcbc7 in term 0.
I20260620 09:30:03.972177 2523 leader_election.cc:302] T 964f3994481c4e9e9d56e8f17dbaa36f P 161eb58fed224099b32195c6e8cfcbc7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 161eb58fed224099b32195c6e8cfcbc7, af9718b2daad486b885084ffb2e5346d; no voters:
I20260620 09:30:03.972355 2661 raft_consensus.cc:2781] T 964f3994481c4e9e9d56e8f17dbaa36f P 161eb58fed224099b32195c6e8cfcbc7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 09:30:03.972419 2661 raft_consensus.cc:482] T 964f3994481c4e9e9d56e8f17dbaa36f P 161eb58fed224099b32195c6e8cfcbc7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 09:30:03.972447 2661 raft_consensus.cc:3037] T 964f3994481c4e9e9d56e8f17dbaa36f P 161eb58fed224099b32195c6e8cfcbc7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:30:03.973135 2661 raft_consensus.cc:504] T 964f3994481c4e9e9d56e8f17dbaa36f P 161eb58fed224099b32195c6e8cfcbc7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af9718b2daad486b885084ffb2e5346d" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 34711 } } peers { permanent_uuid: "50d4531269a9423990ec81398b999df9" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 46619 } } peers { permanent_uuid: "161eb58fed224099b32195c6e8cfcbc7" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 37177 } }
I20260620 09:30:03.973286 2661 leader_election.cc:288] T 964f3994481c4e9e9d56e8f17dbaa36f P 161eb58fed224099b32195c6e8cfcbc7 [CANDIDATE]: Term 1 election: Requested vote from peers af9718b2daad486b885084ffb2e5346d (127.16.43.130:34711), 50d4531269a9423990ec81398b999df9 (127.16.43.129:46619)
I20260620 09:30:03.973465 2322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "964f3994481c4e9e9d56e8f17dbaa36f" candidate_uuid: "161eb58fed224099b32195c6e8cfcbc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50d4531269a9423990ec81398b999df9"
I20260620 09:30:03.973486 2456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "964f3994481c4e9e9d56e8f17dbaa36f" candidate_uuid: "161eb58fed224099b32195c6e8cfcbc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af9718b2daad486b885084ffb2e5346d"
I20260620 09:30:03.973564 2322 raft_consensus.cc:3037] T 964f3994481c4e9e9d56e8f17dbaa36f P 50d4531269a9423990ec81398b999df9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:30:03.973570 2456 raft_consensus.cc:3037] T 964f3994481c4e9e9d56e8f17dbaa36f P af9718b2daad486b885084ffb2e5346d [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:30:03.974315 2322 raft_consensus.cc:2454] T 964f3994481c4e9e9d56e8f17dbaa36f P 50d4531269a9423990ec81398b999df9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 161eb58fed224099b32195c6e8cfcbc7 in term 1.
I20260620 09:30:03.974460 2456 raft_consensus.cc:2454] T 964f3994481c4e9e9d56e8f17dbaa36f P af9718b2daad486b885084ffb2e5346d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 161eb58fed224099b32195c6e8cfcbc7 in term 1.
I20260620 09:30:03.974575 2524 leader_election.cc:302] T 964f3994481c4e9e9d56e8f17dbaa36f P 161eb58fed224099b32195c6e8cfcbc7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 161eb58fed224099b32195c6e8cfcbc7, 50d4531269a9423990ec81398b999df9; no voters:
I20260620 09:30:03.974792 2661 raft_consensus.cc:2781] T 964f3994481c4e9e9d56e8f17dbaa36f P 161eb58fed224099b32195c6e8cfcbc7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:30:03.974974 2661 raft_consensus.cc:686] T 964f3994481c4e9e9d56e8f17dbaa36f P 161eb58fed224099b32195c6e8cfcbc7 [term 1 LEADER]: Becoming Leader. State: Replica: 161eb58fed224099b32195c6e8cfcbc7, State: Running, Role: LEADER
I20260620 09:30:03.975097 2661 consensus_queue.cc:237] T 964f3994481c4e9e9d56e8f17dbaa36f P 161eb58fed224099b32195c6e8cfcbc7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af9718b2daad486b885084ffb2e5346d" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 34711 } } peers { permanent_uuid: "50d4531269a9423990ec81398b999df9" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 46619 } } peers { permanent_uuid: "161eb58fed224099b32195c6e8cfcbc7" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 37177 } }
I20260620 09:30:03.976039 2177 catalog_manager.cc:5495] T 964f3994481c4e9e9d56e8f17dbaa36f P 161eb58fed224099b32195c6e8cfcbc7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 161eb58fed224099b32195c6e8cfcbc7 (127.16.43.131). New cstate: current_term: 1 leader_uuid: "161eb58fed224099b32195c6e8cfcbc7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af9718b2daad486b885084ffb2e5346d" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 34711 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50d4531269a9423990ec81398b999df9" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 46619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "161eb58fed224099b32195c6e8cfcbc7" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 37177 } health_report { overall_health: HEALTHY } } }
I20260620 09:30:04.010516 2651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:04.007895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49858 (local address 127.16.43.131:37177)
0620 09:30:04.007968 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:04.007970 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:04.007995 (+ 25us) server_negotiation.cc:407] Connection header received
0620 09:30:04.008041 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:04.008042 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:04.008070 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:04.008181 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:30:04.008348 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:04.008897 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:04.009697 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:04.009893 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:04.009971 (+ 78us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:04.010137 (+ 166us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:30:04.010230 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:30:04.010346 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:04.010397 (+ 51us) server_negotiation.cc:299] Negotiation successful
0620 09:30:04.010411 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 09:30:04.012792 2456 raft_consensus.cc:1261] T 964f3994481c4e9e9d56e8f17dbaa36f P af9718b2daad486b885084ffb2e5346d [term 1 FOLLOWER]: Refusing update from remote peer 161eb58fed224099b32195c6e8cfcbc7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:30:04.012806 2322 raft_consensus.cc:1261] T 964f3994481c4e9e9d56e8f17dbaa36f P 50d4531269a9423990ec81398b999df9 [term 1 FOLLOWER]: Refusing update from remote peer 161eb58fed224099b32195c6e8cfcbc7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:30:04.013223 2663 consensus_queue.cc:1035] T 964f3994481c4e9e9d56e8f17dbaa36f P 161eb58fed224099b32195c6e8cfcbc7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af9718b2daad486b885084ffb2e5346d" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 34711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:30:04.013363 2661 consensus_queue.cc:1035] T 964f3994481c4e9e9d56e8f17dbaa36f P 161eb58fed224099b32195c6e8cfcbc7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50d4531269a9423990ec81398b999df9" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 46619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:30:04.015575 2666 mvcc.cc:205] Tried to move back new op lower bound from 7298858205232906240 to 7298858205082669056. Current Snapshot: MvccSnapshot[applied={T|T < 7298858205232906240}]
I20260620 09:30:04.016268 2668 mvcc.cc:205] Tried to move back new op lower bound from 7298858205232906240 to 7298858205082669056. Current Snapshot: MvccSnapshot[applied={T|T < 7298858205232906240}]
I20260620 09:30:04.016461 2667 mvcc.cc:205] Tried to move back new op lower bound from 7298858205232906240 to 7298858205082669056. Current Snapshot: MvccSnapshot[applied={T|T < 7298858205232906240}]
I20260620 09:30:04.018682 2176 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44164:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:30:04.018812 2176 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44164:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:30:04.019627 2176 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 25ac048c215b4fd7ab30c13dfba79761: Sending DeleteTablet for 3 replicas of tablet 964f3994481c4e9e9d56e8f17dbaa36f
I20260620 09:30:04.019937 2436 tablet_service.cc:1514] Processing DeleteTablet for tablet 964f3994481c4e9e9d56e8f17dbaa36f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:30:04 UTC) from {username='kudu', principal='kudu/127.16.43.190@KRBTEST.COM'} at 127.0.0.1:43210
I20260620 09:30:04.020009 2302 tablet_service.cc:1514] Processing DeleteTablet for tablet 964f3994481c4e9e9d56e8f17dbaa36f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:30:04 UTC) from {username='kudu', principal='kudu/127.16.43.190@KRBTEST.COM'} at 127.0.0.1:39432
I20260620 09:30:04.020174 2675 tablet_replica.cc:330] T 964f3994481c4e9e9d56e8f17dbaa36f P af9718b2daad486b885084ffb2e5346d: stopping tablet replica
I20260620 09:30:04.020272 2675 raft_consensus.cc:2229] T 964f3994481c4e9e9d56e8f17dbaa36f P af9718b2daad486b885084ffb2e5346d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:30:04.020296 2676 tablet_replica.cc:330] T 964f3994481c4e9e9d56e8f17dbaa36f P 50d4531269a9423990ec81398b999df9: stopping tablet replica
I20260620 09:30:04.020381 2675 raft_consensus.cc:2258] T 964f3994481c4e9e9d56e8f17dbaa36f P af9718b2daad486b885084ffb2e5346d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:30:04.020370 2570 tablet_service.cc:1514] Processing DeleteTablet for tablet 964f3994481c4e9e9d56e8f17dbaa36f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:30:04 UTC) from {username='kudu', principal='kudu/127.16.43.190@KRBTEST.COM'} at 127.0.0.1:49852
I20260620 09:30:04.020532 2676 raft_consensus.cc:2229] T 964f3994481c4e9e9d56e8f17dbaa36f P 50d4531269a9423990ec81398b999df9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:30:04.020650 2677 tablet_replica.cc:330] T 964f3994481c4e9e9d56e8f17dbaa36f P 161eb58fed224099b32195c6e8cfcbc7: stopping tablet replica
I20260620 09:30:04.020735 2677 raft_consensus.cc:2229] T 964f3994481c4e9e9d56e8f17dbaa36f P 161eb58fed224099b32195c6e8cfcbc7 [term 1 LEADER]: Raft consensus shutting down.
I20260620 09:30:04.020846 2677 raft_consensus.cc:2258] T 964f3994481c4e9e9d56e8f17dbaa36f P 161eb58fed224099b32195c6e8cfcbc7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:30:04.020865 2675 ts_tablet_manager.cc:1905] T 964f3994481c4e9e9d56e8f17dbaa36f P af9718b2daad486b885084ffb2e5346d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:30:04.020908 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 2239
I20260620 09:30:04.022297 2677 ts_tablet_manager.cc:1905] T 964f3994481c4e9e9d56e8f17dbaa36f P 161eb58fed224099b32195c6e8cfcbc7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:30:04.024575 2677 ts_tablet_manager.cc:1918] T 964f3994481c4e9e9d56e8f17dbaa36f P 161eb58fed224099b32195c6e8cfcbc7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:30:04.024595 2675 ts_tablet_manager.cc:1918] T 964f3994481c4e9e9d56e8f17dbaa36f P af9718b2daad486b885084ffb2e5346d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:30:04.024660 2677 log.cc:1196] T 964f3994481c4e9e9d56e8f17dbaa36f P 161eb58fed224099b32195c6e8cfcbc7: Deleting WAL directory at /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781947562949448-16558-0/minicluster-data/ts-2/wal/wals/964f3994481c4e9e9d56e8f17dbaa36f
I20260620 09:30:04.024673 2675 log.cc:1196] T 964f3994481c4e9e9d56e8f17dbaa36f P af9718b2daad486b885084ffb2e5346d: Deleting WAL directory at /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781947562949448-16558-0/minicluster-data/ts-1/wal/wals/964f3994481c4e9e9d56e8f17dbaa36f
I20260620 09:30:04.024937 2675 ts_tablet_manager.cc:1939] T 964f3994481c4e9e9d56e8f17dbaa36f P af9718b2daad486b885084ffb2e5346d: Deleting consensus metadata
I20260620 09:30:04.024961 2677 ts_tablet_manager.cc:1939] T 964f3994481c4e9e9d56e8f17dbaa36f P 161eb58fed224099b32195c6e8cfcbc7: Deleting consensus metadata
I20260620 09:30:04.025372 2161 catalog_manager.cc:4841] TS af9718b2daad486b885084ffb2e5346d (127.16.43.130:34711): tablet 964f3994481c4e9e9d56e8f17dbaa36f (table test-table [id=cb62ecb769754c6081a79f6d74e39aa2]) successfully deleted
I20260620 09:30:04.025375 2164 catalog_manager.cc:4841] TS 161eb58fed224099b32195c6e8cfcbc7 (127.16.43.131:37177): tablet 964f3994481c4e9e9d56e8f17dbaa36f (table test-table [id=cb62ecb769754c6081a79f6d74e39aa2]) successfully deleted
W20260620 09:30:04.026731 2162 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.43.129:46619 (error 108)
I20260620 09:30:04.026939 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 2373
I20260620 09:30:04.027824 2648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:04.027691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.129:46619 (local address 127.0.0.1:39448)
0620 09:30:04.027745 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0620 09:30:04.027783 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.129:46619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260620 09:30:04.028033 2162 catalog_manager.cc:4584] TS 50d4531269a9423990ec81398b999df9 (127.16.43.129:46619): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 964f3994481c4e9e9d56e8f17dbaa36f: Network error: Client connection negotiation failed: client connection to 127.16.43.129:46619: connect: Connection refused (error 111)
I20260620 09:30:04.032421 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 2507
I20260620 09:30:04.037427 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 2145
2026-06-20T09:30:04Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2181 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-20T09:30:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:30:04Z Disabled control of system clock
I20260620 09:30:04.066198 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781947562949448-16558-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781947562949448-16558-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781947562949448-16558-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.43.190:40033
--webserver_interface=127.16.43.190
--webserver_port=0
--builtin_ntp_servers=127.16.43.148:43663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.43.190:40033
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 09:30:04.148934 2682 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:30:04.149185 2682 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:30:04.149209 2682 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:30:04.150512 2682 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:30:04.150565 2682 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:30:04.150578 2682 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:30:04.150591 2682 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:30:04.150604 2682 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:30:04.152478 2682 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:43663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781947562949448-16558-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781947562949448-16558-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.43.190:40033
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.43.190:40033
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.43.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2682
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781947562949448-16558-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:30:04.152720 2682 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:30:04.152988 2682 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:30:04.158107 2690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:30:04.158243 2688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:30:04.158326 2682 server_base.cc:1034] running on GCE node
W20260620 09:30:04.158142 2687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:30:04.158696 2682 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:30:04.159003 2682 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:30:04.160228 2682 hybrid_clock.cc:648] HybridClock initialized: now 1781947804160221 us; error 41 us; skew 500 ppm
I20260620 09:30:04.165045 2682 webserver.cc:460] Webserver started at http://127.16.43.190:45971/ using document root <none> and password file <none>
I20260620 09:30:04.165354 2682 fs_manager.cc:359] Metadata directory not provided
I20260620 09:30:04.165432 2682 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:30:04.165596 2682 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:30:04.166720 2682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781947562949448-16558-0/minicluster-data/master-0/data/instance:
uuid: "e7708d1ca8fc4b10943c2a6ad9369dce"
format_stamp: "Formatted at 2026-06-20 09:30:04 on dist-test-slave-706q"
server_key: "bf32f09fcba9435c6179309e77fb96f5"
server_key_iv: "2f3dfb22efa09ebe0c0cccabc8f23b08"
server_key_version: "encryptionkey@0"
I20260620 09:30:04.167095 2682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781947562949448-16558-0/minicluster-data/master-0/wal/instance:
uuid: "e7708d1ca8fc4b10943c2a6ad9369dce"
format_stamp: "Formatted at 2026-06-20 09:30:04 on dist-test-slave-706q"
server_key: "bf32f09fcba9435c6179309e77fb96f5"
server_key_iv: "2f3dfb22efa09ebe0c0cccabc8f23b08"
server_key_version: "encryptionkey@0"
I20260620 09:30:04.168638 2682 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 09:30:04.169512 2696 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:30:04.169836 2682 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:30:04.169955 2682 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781947562949448-16558-0/minicluster-data/master-0/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781947562949448-16558-0/minicluster-data/master-0/wal
uuid: "e7708d1ca8fc4b10943c2a6ad9369dce"
format_stamp: "Formatted at 2026-06-20 09:30:04 on dist-test-slave-706q"
server_key: "bf32f09fcba9435c6179309e77fb96f5"
server_key_iv: "2f3dfb22efa09ebe0c0cccabc8f23b08"
server_key_version: "encryptionkey@0"
I20260620 09:30:04.170040 2682 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781947562949448-16558-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781947562949448-16558-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781947562949448-16558-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:30:04.179510 2682 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:30:04.179946 2682 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:30:04.180107 2682 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:30:04.184105 2682 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.190:40033
I20260620 09:30:04.184142 2748 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.190:40033 every 8 connection(s)
I20260620 09:30:04.184499 2682 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
I20260620 09:30:04.185511 2749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:30:04.187934 2749 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7708d1ca8fc4b10943c2a6ad9369dce: Bootstrap starting.
I20260620 09:30:04.188544 2749 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e7708d1ca8fc4b10943c2a6ad9369dce: Neither blocks nor log segments found. Creating new log.
I20260620 09:30:04.188845 2749 log.cc:824] T 00000000000000000000000000000000 P e7708d1ca8fc4b10943c2a6ad9369dce: Log is configured to *not* fsync() on all Append() calls
I20260620 09:30:04.189510 2749 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7708d1ca8fc4b10943c2a6ad9369dce: No bootstrap required, opened a new log
I20260620 09:30:04.190917 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 2682
I20260620 09:30:04.191145 16558 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781947562949448-16558-0/minicluster-data/master-0/wal/instance
I20260620 09:30:04.191144 2749 raft_consensus.cc:348] T 00000000000000000000000000000000 P e7708d1ca8fc4b10943c2a6ad9369dce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7708d1ca8fc4b10943c2a6ad9369dce" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 40033 } }
I20260620 09:30:04.191340 16558 external_mini_cluster.cc:1428] Setting key 9518dab5e18369764b531ab45dd1bcdf
I20260620 09:30:04.191362 2749 raft_consensus.cc:374] T 00000000000000000000000000000000 P e7708d1ca8fc4b10943c2a6ad9369dce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:30:04.191390 2749 raft_consensus.cc:729] T 00000000000000000000000000000000 P e7708d1ca8fc4b10943c2a6ad9369dce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7708d1ca8fc4b10943c2a6ad9369dce, State: Initialized, Role: FOLLOWER
I20260620 09:30:04.191584 2749 consensus_queue.cc:260] T 00000000000000000000000000000000 P e7708d1ca8fc4b10943c2a6ad9369dce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7708d1ca8fc4b10943c2a6ad9369dce" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 40033 } }
I20260620 09:30:04.191680 2749 raft_consensus.cc:388] T 00000000000000000000000000000000 P e7708d1ca8fc4b10943c2a6ad9369dce [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:30:04.191727 2749 raft_consensus.cc:482] T 00000000000000000000000000000000 P e7708d1ca8fc4b10943c2a6ad9369dce [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:30:04.191774 2749 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e7708d1ca8fc4b10943c2a6ad9369dce [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:30:04.192476 2749 raft_consensus.cc:504] T 00000000000000000000000000000000 P e7708d1ca8fc4b10943c2a6ad9369dce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7708d1ca8fc4b10943c2a6ad9369dce" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 40033 } }
I20260620 09:30:04.192585 2749 leader_election.cc:302] T 00000000000000000000000000000000 P e7708d1ca8fc4b10943c2a6ad9369dce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e7708d1ca8fc4b10943c2a6ad9369dce; no voters:
I20260620 09:30:04.192757 2749 leader_election.cc:288] T 00000000000000000000000000000000 P e7708d1ca8fc4b10943c2a6ad9369dce [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:30:04.192785 2754 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e7708d1ca8fc4b10943c2a6ad9369dce [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:30:04.192894 2754 raft_consensus.cc:686] T 00000000000000000000000000000000 P e7708d1ca8fc4b10943c2a6ad9369dce [term 1 LEADER]: Becoming Leader. State: Replica: e7708d1ca8fc4b10943c2a6ad9369dce, State: Running, Role: LEADER
I20260620 09:30:04.192973 2754 consensus_queue.cc:237] T 00000000000000000000000000000000 P e7708d1ca8fc4b10943c2a6ad9369dce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7708d1ca8fc4b10943c2a6ad9369dce" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 40033 } }
I20260620 09:30:04.193017 2749 sys_catalog.cc:564] T 00000000000000000000000000000000 P e7708d1ca8fc4b10943c2a6ad9369dce [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:30:04.193320 2754 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7708d1ca8fc4b10943c2a6ad9369dce [sys.catalog]: SysCatalogTable state changed. Reason: New leader e7708d1ca8fc4b10943c2a6ad9369dce. Latest consensus state: current_term: 1 leader_uuid: "e7708d1ca8fc4b10943c2a6ad9369dce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7708d1ca8fc4b10943c2a6ad9369dce" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 40033 } } }
I20260620 09:30:04.193389 2754 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7708d1ca8fc4b10943c2a6ad9369dce [sys.catalog]: This master's current role is: LEADER
I20260620 09:30:04.193338 2756 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7708d1ca8fc4b10943c2a6ad9369dce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e7708d1ca8fc4b10943c2a6ad9369dce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7708d1ca8fc4b10943c2a6ad9369dce" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 40033 } } }
I20260620 09:30:04.193593 2756 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7708d1ca8fc4b10943c2a6ad9369dce [sys.catalog]: This master's current role is: LEADER
W20260620 09:30:04.194775 2770 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e7708d1ca8fc4b10943c2a6ad9369dce: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 09:30:04.194845 2770 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 09:30:04.194908 2759 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:30:04.195075 2759 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:30:04.196740 2759 catalog_manager.cc:1347] Generated new cluster ID: 29466e08ef5d4e1097083bf8765d23a0
I20260620 09:30:04.196797 2759 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:30:04.198050 2753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:04.191884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52686 (local address 127.16.43.190:40033)
0620 09:30:04.192147 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:04.192156 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:04.192189 (+ 33us) server_negotiation.cc:407] Connection header received
0620 09:30:04.192312 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:04.192320 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:04.192403 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:04.192557 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:30:04.192950 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:04.194736 (+ 1786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:04.195934 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:04.196467 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:04.196617 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:04.197386 (+ 769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:30:04.197390 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:30:04.197390 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:30:04.197577 (+ 187us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:30:04.197614 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:30:04.197625 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:30:04.197722 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:04.197801 (+ 79us) server_negotiation.cc:299] Negotiation successful
0620 09:30:04.197882 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":118,"threads_started":1}
I20260620 09:30:04.204769 2759 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:30:04.205303 2759 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:30:04.213495 2759 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e7708d1ca8fc4b10943c2a6ad9369dce: Generated new TSK 0
I20260620 09:30:04.213711 2759 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 09:30:04.215776 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781947562949448-16558-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781947562949448-16558-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781947562949448-16558-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781947562949448-16558-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.43.129:0
--local_ip_for_outbound_sockets=127.16.43.129
--webserver_interface=127.16.43.129
--webserver_port=0
--tserver_master_addrs=127.16.43.190:40033
--builtin_ntp_servers=127.16.43.148:43663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 09:30:04.308154 2773 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:30:04.308385 2773 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:30:04.308411 2773 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:30:04.309660 2773 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:30:04.309712 2773 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:30:04.309741 2773 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.43.129
I20260620 09:30:04.311223 2773 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:43663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781947562949448-16558-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781947562949448-16558-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.43.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781947562949448-16558-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.43.129
--webserver_port=0
--tserver_master_addrs=127.16.43.190:40033
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2773
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.43.129
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781947562949448-16558-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:30:04.311468 2773 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:30:04.311828 2773 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:30:04.317937 2779 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:30:04.317932 2778 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:30:04.318194 2781 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:30:04.318251 2773 server_base.cc:1034] running on GCE node
I20260620 09:30:04.318461 2773 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:30:04.318770 2773 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:30:04.319917 2773 hybrid_clock.cc:648] HybridClock initialized: now 1781947804319898 us; error 34 us; skew 500 ppm
I20260620 09:30:04.321338 2773 webserver.cc:460] Webserver started at http://127.16.43.129:41581/ using document root <none> and password file <none>
I20260620 09:30:04.321643 2773 fs_manager.cc:359] Metadata directory not provided
I20260620 09:30:04.321692 2773 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:30:04.321833 2773 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:30:04.322973 2773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781947562949448-16558-0/minicluster-data/ts-0/data/instance:
uuid: "2397eb6cc16a487484a50bf707518254"
format_stamp: "Formatted at 2026-06-20 09:30:04 on dist-test-slave-706q"
server_key: "ce0bef516d3ab61ab3598c331e077f3b"
server_key_iv: "396a700e9be0f8086f1b9611d18104e2"
server_key_version: "encryptionkey@0"
I20260620 09:30:04.323359 2773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781947562949448-16558-0/minicluster-data/ts-0/wal/instance:
uuid: "2397eb6cc16a487484a50bf707518254"
format_stamp: "Formatted at 2026-06-20 09:30:04 on dist-test-slave-706q"
server_key: "ce0bef516d3ab61ab3598c331e077f3b"
server_key_iv: "396a700e9be0f8086f1b9611d18104e2"
server_key_version: "encryptionkey@0"
I20260620 09:30:04.324815 2773 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:30:04.325642 2787 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:30:04.325801 2773 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:30:04.325879 2773 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781947562949448-16558-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781947562949448-16558-0/minicluster-data/ts-0/wal
uuid: "2397eb6cc16a487484a50bf707518254"
format_stamp: "Formatted at 2026-06-20 09:30:04 on dist-test-slave-706q"
server_key: "ce0bef516d3ab61ab3598c331e077f3b"
server_key_iv: "396a700e9be0f8086f1b9611d18104e2"
server_key_version: "encryptionkey@0"
I20260620 09:30:04.325953 2773 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781947562949448-16558-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781947562949448-16558-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781947562949448-16558-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:30:04.346225 2773 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:30:04.346524 2773 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:30:04.346634 2773 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:30:04.346832 2773 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:30:04.347209 2773 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:30:04.347252 2773 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:30:04.347276 2773 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:30:04.347327 2773 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:30:04.353097 2773 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.129:41595
I20260620 09:30:04.353164 2900 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.129:41595 every 8 connection(s)
I20260620 09:30:04.353536 2773 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781947562949448-16558-0/minicluster-data/ts-0/data/info.pb
I20260620 09:30:04.359232 2753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:04.354429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.129:50925 (local address 127.16.43.190:40033)
0620 09:30:04.354499 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:04.354502 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:04.354813 (+ 311us) server_negotiation.cc:407] Connection header received
0620 09:30:04.355323 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:04.355324 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:04.355338 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:04.355410 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:30:04.356173 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:04.356689 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:04.357691 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:04.357931 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:04.357992 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:04.358945 (+ 953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:30:04.358946 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:30:04.358946 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:30:04.358951 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:30:04.358968 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:30:04.358972 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:30:04.359023 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:04.359130 (+ 107us) server_negotiation.cc:299] Negotiation successful
0620 09:30:04.359166 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 09:30:04.359467 2903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:04.354480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:40033 (local address 127.16.43.129:50925)
0620 09:30:04.354669 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0620 09:30:04.354716 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 09:30:04.355210 (+ 494us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:30:04.355472 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:30:04.355475 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:30:04.355484 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:30:04.356067 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:30:04.356073 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:30:04.356781 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:30:04.356782 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:30:04.357579 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:30:04.357581 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:30:04.358022 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:30:04.358023 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:30:04.358140 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:04.358795 (+ 655us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 09:30:04.358796 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:30:04.358824 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 09:30:04.358848 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:30:04.359044 (+ 196us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:30:04.359053 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:30:04.359056 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 09:30:04.359115 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 09:30:04.359217 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":83,"threads_started":1}
I20260620 09:30:04.360088 2901 heartbeater.cc:347] Connected to a master server at 127.16.43.190:40033
I20260620 09:30:04.360200 2901 heartbeater.cc:464] Registering TS with master...
I20260620 09:30:04.360358 2901 heartbeater.cc:511] Master 127.16.43.190:40033 requested a full tablet report, sending...
I20260620 09:30:04.360868 2713 ts_manager.cc:194] Registered new tserver with Master: 2397eb6cc16a487484a50bf707518254 (127.16.43.129:41595)
I20260620 09:30:04.361461 2713 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.43.129:50925
I20260620 09:30:04.361835 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 2773
I20260620 09:30:04.361924 16558 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781947562949448-16558-0/minicluster-data/ts-0/wal/instance
I20260620 09:30:04.362138 16558 external_mini_cluster.cc:1428] Setting key e421c57b47109c309973a619342d5511
I20260620 09:30:04.363128 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781947562949448-16558-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781947562949448-16558-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781947562949448-16558-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781947562949448-16558-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.43.130:0
--local_ip_for_outbound_sockets=127.16.43.130
--webserver_interface=127.16.43.130
--webserver_port=0
--tserver_master_addrs=127.16.43.190:40033
--builtin_ntp_servers=127.16.43.148:43663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 09:30:04.450502 2904 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:30:04.450767 2904 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:30:04.450809 2904 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:30:04.452122 2904 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:30:04.452191 2904 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:30:04.452248 2904 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.43.130
I20260620 09:30:04.453688 2904 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:43663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781947562949448-16558-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781947562949448-16558-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.43.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781947562949448-16558-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.43.130
--webserver_port=0
--tserver_master_addrs=127.16.43.190:40033
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2904
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.43.130
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781947562949448-16558-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:30:04.453984 2904 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:30:04.454317 2904 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 09:30:04.460009 2904 server_base.cc:1034] running on GCE node
W20260620 09:30:04.460009 2912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:30:04.460044 2910 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:30:04.460155 2909 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:30:04.460376 2904 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:30:04.460649 2904 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:30:04.461813 2904 hybrid_clock.cc:648] HybridClock initialized: now 1781947804461802 us; error 30 us; skew 500 ppm
I20260620 09:30:04.463189 2904 webserver.cc:460] Webserver started at http://127.16.43.130:41943/ using document root <none> and password file <none>
I20260620 09:30:04.463485 2904 fs_manager.cc:359] Metadata directory not provided
I20260620 09:30:04.463588 2904 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:30:04.463752 2904 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:30:04.464854 2904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781947562949448-16558-0/minicluster-data/ts-1/data/instance:
uuid: "f78e6d1fa6c24f458715cda8b66dd2cf"
format_stamp: "Formatted at 2026-06-20 09:30:04 on dist-test-slave-706q"
server_key: "b378f0ed25fdfec503dc67d40d30743d"
server_key_iv: "76ab47f46691a974d4aeb6fb7b6ec416"
server_key_version: "encryptionkey@0"
I20260620 09:30:04.465225 2904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781947562949448-16558-0/minicluster-data/ts-1/wal/instance:
uuid: "f78e6d1fa6c24f458715cda8b66dd2cf"
format_stamp: "Formatted at 2026-06-20 09:30:04 on dist-test-slave-706q"
server_key: "b378f0ed25fdfec503dc67d40d30743d"
server_key_iv: "76ab47f46691a974d4aeb6fb7b6ec416"
server_key_version: "encryptionkey@0"
I20260620 09:30:04.466651 2904 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 09:30:04.467648 2918 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:30:04.467907 2904 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:30:04.468019 2904 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781947562949448-16558-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781947562949448-16558-0/minicluster-data/ts-1/wal
uuid: "f78e6d1fa6c24f458715cda8b66dd2cf"
format_stamp: "Formatted at 2026-06-20 09:30:04 on dist-test-slave-706q"
server_key: "b378f0ed25fdfec503dc67d40d30743d"
server_key_iv: "76ab47f46691a974d4aeb6fb7b6ec416"
server_key_version: "encryptionkey@0"
I20260620 09:30:04.468108 2904 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781947562949448-16558-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781947562949448-16558-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781947562949448-16558-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:30:04.484365 2904 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:30:04.484740 2904 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:30:04.484898 2904 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:30:04.485140 2904 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:30:04.485474 2904 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:30:04.485527 2904 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:30:04.485569 2904 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:30:04.485591 2904 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:30:04.491745 2904 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.130:44489
I20260620 09:30:04.491786 3031 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.130:44489 every 8 connection(s)
I20260620 09:30:04.492136 2904 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781947562949448-16558-0/minicluster-data/ts-1/data/info.pb
I20260620 09:30:04.497933 2753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:04.492897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.130:49687 (local address 127.16.43.190:40033)
0620 09:30:04.492978 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:04.492980 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:04.493257 (+ 277us) server_negotiation.cc:407] Connection header received
0620 09:30:04.493845 (+ 588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:04.493846 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:04.493857 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:04.493922 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:30:04.494713 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:04.495323 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:04.496389 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:04.496598 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:04.496666 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:04.497642 (+ 976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:30:04.497643 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:30:04.497644 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:30:04.497649 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:30:04.497664 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:30:04.497669 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:30:04.497717 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:04.497842 (+ 125us) server_negotiation.cc:299] Negotiation successful
0620 09:30:04.497869 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 09:30:04.498178 3034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:04.492880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:40033 (local address 127.16.43.130:49687)
0620 09:30:04.493090 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0620 09:30:04.493165 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0620 09:30:04.493715 (+ 550us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:30:04.493996 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:30:04.493999 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:30:04.494008 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:30:04.494582 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:30:04.494587 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:30:04.495422 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:30:04.495423 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:30:04.496295 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:30:04.496298 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:30:04.496731 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:30:04.496732 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:30:04.496856 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:04.497437 (+ 581us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 09:30:04.497438 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:30:04.497468 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 09:30:04.497492 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:30:04.497752 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:30:04.497759 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:30:04.497762 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 09:30:04.497837 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 09:30:04.497922 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":114,"threads_started":1}
I20260620 09:30:04.498708 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 2904
I20260620 09:30:04.498834 16558 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781947562949448-16558-0/minicluster-data/ts-1/wal/instance
I20260620 09:30:04.498873 3032 heartbeater.cc:347] Connected to a master server at 127.16.43.190:40033
I20260620 09:30:04.498971 3032 heartbeater.cc:464] Registering TS with master...
I20260620 09:30:04.499058 16558 external_mini_cluster.cc:1428] Setting key 9952dac70fd7d4ef29f64dfe271a5e17
I20260620 09:30:04.499212 3032 heartbeater.cc:511] Master 127.16.43.190:40033 requested a full tablet report, sending...
I20260620 09:30:04.499693 2712 ts_manager.cc:194] Registered new tserver with Master: f78e6d1fa6c24f458715cda8b66dd2cf (127.16.43.130:44489)
I20260620 09:30:04.500017 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781947562949448-16558-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781947562949448-16558-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781947562949448-16558-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781947562949448-16558-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.43.131:0
--local_ip_for_outbound_sockets=127.16.43.131
--webserver_interface=127.16.43.131
--webserver_port=0
--tserver_master_addrs=127.16.43.190:40033
--builtin_ntp_servers=127.16.43.148:43663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 09:30:04.500103 2712 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.43.130:49687
W20260620 09:30:04.581611 3035 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:30:04.581815 3035 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:30:04.581838 3035 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:30:04.582994 3035 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:30:04.583037 3035 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:30:04.583062 3035 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.43.131
I20260620 09:30:04.584523 3035 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:43663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781947562949448-16558-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781947562949448-16558-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.43.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781947562949448-16558-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.43.131
--webserver_port=0
--tserver_master_addrs=127.16.43.190:40033
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3035
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.43.131
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781947562949448-16558-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:30:04.584786 3035 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:30:04.585096 3035 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:30:04.590246 3043 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:30:04.590255 3040 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:30:04.590327 3041 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:30:04.590360 3035 server_base.cc:1034] running on GCE node
I20260620 09:30:04.590750 3035 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:30:04.591043 3035 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:30:04.592206 3035 hybrid_clock.cc:648] HybridClock initialized: now 1781947804592186 us; error 35 us; skew 500 ppm
I20260620 09:30:04.593730 3035 webserver.cc:460] Webserver started at http://127.16.43.131:39247/ using document root <none> and password file <none>
I20260620 09:30:04.593972 3035 fs_manager.cc:359] Metadata directory not provided
I20260620 09:30:04.594043 3035 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:30:04.594192 3035 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:30:04.595290 3035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781947562949448-16558-0/minicluster-data/ts-2/data/instance:
uuid: "7d9426b8224c4426bd1fcb52ff033f13"
format_stamp: "Formatted at 2026-06-20 09:30:04 on dist-test-slave-706q"
server_key: "16b185a3696617fe13fd17f1b69e3eb3"
server_key_iv: "02bc701a263e19e889a13baaf48a65cf"
server_key_version: "encryptionkey@0"
I20260620 09:30:04.595705 3035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781947562949448-16558-0/minicluster-data/ts-2/wal/instance:
uuid: "7d9426b8224c4426bd1fcb52ff033f13"
format_stamp: "Formatted at 2026-06-20 09:30:04 on dist-test-slave-706q"
server_key: "16b185a3696617fe13fd17f1b69e3eb3"
server_key_iv: "02bc701a263e19e889a13baaf48a65cf"
server_key_version: "encryptionkey@0"
I20260620 09:30:04.597023 3035 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 09:30:04.597831 3049 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:30:04.598027 3035 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 09:30:04.598124 3035 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781947562949448-16558-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781947562949448-16558-0/minicluster-data/ts-2/wal
uuid: "7d9426b8224c4426bd1fcb52ff033f13"
format_stamp: "Formatted at 2026-06-20 09:30:04 on dist-test-slave-706q"
server_key: "16b185a3696617fe13fd17f1b69e3eb3"
server_key_iv: "02bc701a263e19e889a13baaf48a65cf"
server_key_version: "encryptionkey@0"
I20260620 09:30:04.598208 3035 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781947562949448-16558-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781947562949448-16558-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781947562949448-16558-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:30:04.617537 3035 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:30:04.617892 3035 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:30:04.618052 3035 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:30:04.618301 3035 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:30:04.618654 3035 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:30:04.618723 3035 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:30:04.618767 3035 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:30:04.618813 3035 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:30:04.624395 3035 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.131:33297
I20260620 09:30:04.624480 3162 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.131:33297 every 8 connection(s)
I20260620 09:30:04.624828 3035 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781947562949448-16558-0/minicluster-data/ts-2/data/info.pb
I20260620 09:30:04.625061 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 3035
I20260620 09:30:04.625180 16558 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781947562949448-16558-0/minicluster-data/ts-2/wal/instance
I20260620 09:30:04.625394 16558 external_mini_cluster.cc:1428] Setting key 3c9baf89434c3dd439d73ddb9cb41499
I20260620 09:30:04.630393 2753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:04.625656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.131:41691 (local address 127.16.43.190:40033)
0620 09:30:04.625716 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:04.625718 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:04.626027 (+ 309us) server_negotiation.cc:407] Connection header received
0620 09:30:04.626626 (+ 599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:04.626626 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:04.626636 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:04.626683 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:30:04.627318 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:04.627907 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:04.629017 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:04.629244 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:04.629298 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:04.630127 (+ 829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:30:04.630129 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:30:04.630129 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:30:04.630134 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:30:04.630150 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:30:04.630154 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:30:04.630201 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:04.630298 (+ 97us) server_negotiation.cc:299] Negotiation successful
0620 09:30:04.630327 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 09:30:04.630545 3165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:04.625678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:40033 (local address 127.16.43.131:41691)
0620 09:30:04.625880 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0620 09:30:04.625926 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 09:30:04.626527 (+ 601us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:30:04.626737 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:30:04.626740 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:30:04.626748 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:30:04.627249 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:30:04.627253 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:30:04.627988 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:30:04.627989 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:30:04.628930 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:30:04.628933 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:30:04.629318 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:30:04.629319 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:30:04.629429 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:04.629978 (+ 549us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 09:30:04.629978 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:30:04.630018 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 09:30:04.630042 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:30:04.630212 (+ 170us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:30:04.630216 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:30:04.630218 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 09:30:04.630271 (+ 53us) client_negotiation.cc:240] Negotiation successful
0620 09:30:04.630352 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":97,"threads_started":1}
I20260620 09:30:04.631073 3163 heartbeater.cc:347] Connected to a master server at 127.16.43.190:40033
I20260620 09:30:04.631178 3163 heartbeater.cc:464] Registering TS with master...
I20260620 09:30:04.631372 3163 heartbeater.cc:511] Master 127.16.43.190:40033 requested a full tablet report, sending...
I20260620 09:30:04.631860 2712 ts_manager.cc:194] Registered new tserver with Master: 7d9426b8224c4426bd1fcb52ff033f13 (127.16.43.131:33297)
I20260620 09:30:04.632246 2712 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.43.131:41691
I20260620 09:30:04.637125 16558 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 09:30:04.641799 2753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:04.638524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52696 (local address 127.16.43.190:40033)
0620 09:30:04.638678 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:04.638680 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:04.638692 (+ 12us) server_negotiation.cc:407] Connection header received
0620 09:30:04.638806 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:04.638807 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:04.638820 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:04.638866 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:30:04.639046 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:04.639606 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:04.640368 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:04.640594 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:04.640677 (+ 83us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:04.641414 (+ 737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:30:04.641415 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:30:04.641416 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:30:04.641421 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:30:04.641437 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:30:04.641441 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:30:04.641561 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:04.641688 (+ 127us) server_negotiation.cc:299] Negotiation successful
0620 09:30:04.641702 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127}
I20260620 09:30:04.644034 2712 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52696:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 09:30:04.645668 2712 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 09:30:04.652326 3172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:04.648104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.131:33297 (local address 127.0.0.1:34734)
0620 09:30:04.648309 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0620 09:30:04.648318 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 09:30:04.648565 (+ 247us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:30:04.648860 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:30:04.648864 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:30:04.648866 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:30:04.648941 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:30:04.648943 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:30:04.649943 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:30:04.649944 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:30:04.651140 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:30:04.651142 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:30:04.652025 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:30:04.652026 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:30:04.652101 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:04.652114 (+ 13us) client_negotiation.cc:769] Sending connection context
0620 09:30:04.652192 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 09:30:04.652247 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":106,"threads_started":1}
I20260620 09:30:04.652338 3175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:04.648397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34734 (local address 127.16.43.131:33297)
0620 09:30:04.648616 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:04.648621 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:04.648631 (+ 10us) server_negotiation.cc:407] Connection header received
0620 09:30:04.648673 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:04.648676 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:04.648692 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:04.648769 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:30:04.649025 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:04.649851 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:04.651235 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:04.651938 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:04.652026 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:04.652121 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:04.652206 (+ 85us) server_negotiation.cc:299] Negotiation successful
0620 09:30:04.652246 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":111,"threads_started":1}
I20260620 09:30:04.652542 3171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:04.648101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.130:44489 (local address 127.0.0.1:46948)
0620 09:30:04.648240 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 09:30:04.648300 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0620 09:30:04.648376 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:30:04.648966 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:30:04.648967 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:30:04.648969 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:30:04.649037 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:30:04.649039 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:30:04.650362 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:30:04.650363 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:30:04.651357 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:30:04.651358 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:30:04.652315 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:30:04.652316 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:30:04.652378 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:04.652382 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 09:30:04.652414 (+ 32us) client_negotiation.cc:240] Negotiation successful
0620 09:30:04.652464 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"mutex_wait_us":1,"thread_start_us":118,"threads_started":1}
I20260620 09:30:04.652797 3173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:04.648101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.129:41595 (local address 127.0.0.1:42420)
0620 09:30:04.648869 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0620 09:30:04.648879 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 09:30:04.648958 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:30:04.649208 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:30:04.649209 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:30:04.649211 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:30:04.649248 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:30:04.649249 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:30:04.650520 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:30:04.650521 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:30:04.651592 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:30:04.651595 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:30:04.652578 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:30:04.652579 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:30:04.652633 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:04.652636 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 09:30:04.652694 (+ 58us) client_negotiation.cc:240] Negotiation successful
0620 09:30:04.652734 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":753,"mutex_wait_us":4,"thread_start_us":77,"threads_started":1}
I20260620 09:30:04.652869 3174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:04.648382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42420 (local address 127.16.43.129:41595)
0620 09:30:04.648621 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:04.648626 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:04.648931 (+ 305us) server_negotiation.cc:407] Connection header received
0620 09:30:04.649051 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:04.649054 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:04.649070 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:04.649121 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:30:04.649331 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:04.650434 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:04.651677 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:04.652408 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:04.652467 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:04.652545 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:04.652729 (+ 184us) server_negotiation.cc:299] Negotiation successful
0620 09:30:04.652782 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":88,"threads_started":1}
I20260620 09:30:04.652881 3176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:04.648614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46948 (local address 127.16.43.130:44489)
0620 09:30:04.648775 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:04.648779 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:04.648788 (+ 9us) server_negotiation.cc:407] Connection header received
0620 09:30:04.648833 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:04.648837 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:04.648852 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:04.649122 (+ 270us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:30:04.649171 (+ 49us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:04.650270 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:04.651443 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:04.652206 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:04.652617 (+ 411us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:04.652726 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:04.652751 (+ 25us) server_negotiation.cc:299] Negotiation successful
0620 09:30:04.652791 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":88,"threads_started":1}
I20260620 09:30:04.653541 2966 tablet_service.cc:1467] Processing CreateTablet for tablet 7c4a7abb8dc44234b92f1b1f40f9f3e9 (DEFAULT_TABLE table=test-table [id=6f6689203415451e8e28a3094ec2abbe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:30:04.653725 2835 tablet_service.cc:1467] Processing CreateTablet for tablet 7c4a7abb8dc44234b92f1b1f40f9f3e9 (DEFAULT_TABLE table=test-table [id=6f6689203415451e8e28a3094ec2abbe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:30:04.653888 2966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c4a7abb8dc44234b92f1b1f40f9f3e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:30:04.654069 2835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c4a7abb8dc44234b92f1b1f40f9f3e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:30:04.656870 3177 tablet_bootstrap.cc:492] T 7c4a7abb8dc44234b92f1b1f40f9f3e9 P 2397eb6cc16a487484a50bf707518254: Bootstrap starting.
I20260620 09:30:04.657125 3178 tablet_bootstrap.cc:492] T 7c4a7abb8dc44234b92f1b1f40f9f3e9 P f78e6d1fa6c24f458715cda8b66dd2cf: Bootstrap starting.
I20260620 09:30:04.657618 3177 tablet_bootstrap.cc:654] T 7c4a7abb8dc44234b92f1b1f40f9f3e9 P 2397eb6cc16a487484a50bf707518254: Neither blocks nor log segments found. Creating new log.
I20260620 09:30:04.657727 3178 tablet_bootstrap.cc:654] T 7c4a7abb8dc44234b92f1b1f40f9f3e9 P f78e6d1fa6c24f458715cda8b66dd2cf: Neither blocks nor log segments found. Creating new log.
I20260620 09:30:04.657939 3177 log.cc:824] T 7c4a7abb8dc44234b92f1b1f40f9f3e9 P 2397eb6cc16a487484a50bf707518254: Log is configured to *not* fsync() on all Append() calls
I20260620 09:30:04.657821 3097 tablet_service.cc:1467] Processing CreateTablet for tablet 7c4a7abb8dc44234b92f1b1f40f9f3e9 (DEFAULT_TABLE table=test-table [id=6f6689203415451e8e28a3094ec2abbe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:30:04.658023 3178 log.cc:824] T 7c4a7abb8dc44234b92f1b1f40f9f3e9 P f78e6d1fa6c24f458715cda8b66dd2cf: Log is configured to *not* fsync() on all Append() calls
I20260620 09:30:04.658176 3097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c4a7abb8dc44234b92f1b1f40f9f3e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:30:04.658737 3177 tablet_bootstrap.cc:492] T 7c4a7abb8dc44234b92f1b1f40f9f3e9 P 2397eb6cc16a487484a50bf707518254: No bootstrap required, opened a new log
I20260620 09:30:04.658814 3177 ts_tablet_manager.cc:1397] T 7c4a7abb8dc44234b92f1b1f40f9f3e9 P 2397eb6cc16a487484a50bf707518254: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 09:30:04.659344 3178 tablet_bootstrap.cc:492] T 7c4a7abb8dc44234b92f1b1f40f9f3e9 P f78e6d1fa6c24f458715cda8b66dd2cf: No bootstrap required, opened a new log
I20260620 09:30:04.659428 3178 ts_tablet_manager.cc:1397] T 7c4a7abb8dc44234b92f1b1f40f9f3e9 P f78e6d1fa6c24f458715cda8b66dd2cf: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 09:30:04.661370 3178 raft_consensus.cc:348] T 7c4a7abb8dc44234b92f1b1f40f9f3e9 P f78e6d1fa6c24f458715cda8b66dd2cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2397eb6cc16a487484a50bf707518254" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 41595 } } peers { permanent_uuid: "7d9426b8224c4426bd1fcb52ff033f13" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 33297 } } peers { permanent_uuid: "f78e6d1fa6c24f458715cda8b66dd2cf" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 44489 } }
I20260620 09:30:04.661532 3178 raft_consensus.cc:374] T 7c4a7abb8dc44234b92f1b1f40f9f3e9 P f78e6d1fa6c24f458715cda8b66dd2cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:30:04.661576 3178 raft_consensus.cc:729] T 7c4a7abb8dc44234b92f1b1f40f9f3e9 P f78e6d1fa6c24f458715cda8b66dd2cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f78e6d1fa6c24f458715cda8b66dd2cf, State: Initialized, Role: FOLLOWER
I20260620 09:30:04.661685 3178 consensus_queue.cc:260] T 7c4a7abb8dc44234b92f1b1f40f9f3e9 P f78e6d1fa6c24f458715cda8b66dd2cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2397eb6cc16a487484a50bf707518254" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 41595 } } peers { permanent_uuid: "7d9426b8224c4426bd1fcb52ff033f13" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 33297 } } peers { permanent_uuid: "f78e6d1fa6c24f458715cda8b66dd2cf" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 44489 } }
I20260620 09:30:04.661659 3177 raft_consensus.cc:348] T 7c4a7abb8dc44234b92f1b1f40f9f3e9 P 2397eb6cc16a487484a50bf707518254 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2397eb6cc16a487484a50bf707518254" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 41595 } } peers { permanent_uuid: "7d9426b8224c4426bd1fcb52ff033f13" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 33297 } } peers { permanent_uuid: "f78e6d1fa6c24f458715cda8b66dd2cf" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 44489 } }
I20260620 09:30:04.661823 3177 raft_consensus.cc:374] T 7c4a7abb8dc44234b92f1b1f40f9f3e9 P 2397eb6cc16a487484a50bf707518254 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:30:04.661865 3177 raft_consensus.cc:729] T 7c4a7abb8dc44234b92f1b1f40f9f3e9 P 2397eb6cc16a487484a50bf707518254 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2397eb6cc16a487484a50bf707518254, State: Initialized, Role: FOLLOWER
I20260620 09:30:04.661953 3178 ts_tablet_manager.cc:1428] T 7c4a7abb8dc44234b92f1b1f40f9f3e9 P f78e6d1fa6c24f458715cda8b66dd2cf: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 09:30:04.662076 3181 tablet_bootstrap.cc:492] T 7c4a7abb8dc44234b92f1b1f40f9f3e9 P 7d9426b8224c4426bd1fcb52ff033f13: Bootstrap starting.
I20260620 09:30:04.662022 3177 consensus_queue.cc:260] T 7c4a7abb8dc44234b92f1b1f40f9f3e9 P 2397eb6cc16a487484a50bf707518254 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2397eb6cc16a487484a50bf707518254" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 41595 } } peers { permanent_uuid: "7d9426b8224c4426bd1fcb52ff033f13" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 33297 } } peers { permanent_uuid: "f78e6d1fa6c24f458715cda8b66dd2cf" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 44489 } }
I20260620 09:30:04.662213 3032 heartbeater.cc:503] Master 127.16.43.190:40033 was elected leader, sending a full tablet report...
I20260620 09:30:04.662453 3177 ts_tablet_manager.cc:1428] T 7c4a7abb8dc44234b92f1b1f40f9f3e9 P 2397eb6cc16a487484a50bf707518254: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260620 09:30:04.662494 2901 heartbeater.cc:503] Master 127.16.43.190:40033 was elected leader, sending a full tablet report...
I20260620 09:30:04.662781 3181 tablet_bootstrap.cc:654] T 7c4a7abb8dc44234b92f1b1f40f9f3e9 P 7d9426b8224c4426bd1fcb52ff033f13: Neither blocks nor log segments found. Creating new log.
I20260620 09:30:04.663116 3181 log.cc:824] T 7c4a7abb8dc44234b92f1b1f40f9f3e9 P 7d9426b8224c4426bd1fcb52ff033f13: Log is configured to *not* fsync() on all Append() calls
I20260620 09:30:04.664064 3181 tablet_bootstrap.cc:492] T 7c4a7abb8dc44234b92f1b1f40f9f3e9 P 7d9426b8224c4426bd1fcb52ff033f13: No bootstrap required, opened a new log
I20260620 09:30:04.664116 3181 ts_tablet_manager.cc:1397] T 7c4a7abb8dc44234b92f1b1f40f9f3e9 P 7d9426b8224c4426bd1fcb52ff033f13: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 09:30:04.665921 3181 raft_consensus.cc:348] T 7c4a7abb8dc44234b92f1b1f40f9f3e9 P 7d9426b8224c4426bd1fcb52ff033f13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2397eb6cc16a487484a50bf707518254" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 41595 } } peers { permanent_uuid: "7d9426b8224c4426bd1fcb52ff033f13" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 33297 } } peers { permanent_uuid: "f78e6d1fa6c24f458715cda8b66dd2cf" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 44489 } }
I20260620 09:30:04.666049 3181 raft_consensus.cc:374] T 7c4a7abb8dc44234b92f1b1f40f9f3e9 P 7d9426b8224c4426bd1fcb52ff033f13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:30:04.666069 3181 raft_consensus.cc:729] T 7c4a7abb8dc44234b92f1b1f40f9f3e9 P 7d9426b8224c4426bd1fcb52ff033f13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d9426b8224c4426bd1fcb52ff033f13, State: Initialized, Role: FOLLOWER
I20260620 09:30:04.666178 3181 consensus_queue.cc:260] T 7c4a7abb8dc44234b92f1b1f40f9f3e9 P 7d9426b8224c4426bd1fcb52ff033f13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2397eb6cc16a487484a50bf707518254" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 41595 } } peers { permanent_uuid: "7d9426b8224c4426bd1fcb52ff033f13" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 33297 } } peers { permanent_uuid: "f78e6d1fa6c24f458715cda8b66dd2cf" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 44489 } }
I20260620 09:30:04.666424 3181 ts_tablet_manager.cc:1428] T 7c4a7abb8dc44234b92f1b1f40f9f3e9 P 7d9426b8224c4426bd1fcb52ff033f13: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 09:30:04.666486 3163 heartbeater.cc:503] Master 127.16.43.190:40033 was elected leader, sending a full tablet report...
I20260620 09:30:04.709381 3183 raft_consensus.cc:482] T 7c4a7abb8dc44234b92f1b1f40f9f3e9 P 2397eb6cc16a487484a50bf707518254 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:30:04.709529 3183 raft_consensus.cc:504] T 7c4a7abb8dc44234b92f1b1f40f9f3e9 P 2397eb6cc16a487484a50bf707518254 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2397eb6cc16a487484a50bf707518254" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 41595 } } peers { permanent_uuid: "7d9426b8224c4426bd1fcb52ff033f13" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 33297 } } peers { permanent_uuid: "f78e6d1fa6c24f458715cda8b66dd2cf" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 44489 } }
I20260620 09:30:04.709928 3183 leader_election.cc:288] T 7c4a7abb8dc44234b92f1b1f40f9f3e9 P 2397eb6cc16a487484a50bf707518254 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7d9426b8224c4426bd1fcb52ff033f13 (127.16.43.131:33297), f78e6d1fa6c24f458715cda8b66dd2cf (127.16.43.130:44489)
I20260620 09:30:04.712962 2903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:04.710099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.131:33297 (local address 127.16.43.129:34655)
0620 09:30:04.710185 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0620 09:30:04.710197 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 09:30:04.710242 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:30:04.710568 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:30:04.710568 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:30:04.710569 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:30:04.710628 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:30:04.710629 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:30:04.711282 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:30:04.711283 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:30:04.712145 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:30:04.712148 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:30:04.712688 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:30:04.712689 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:30:04.712766 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:04.712770 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 09:30:04.712818 (+ 48us) client_negotiation.cc:240] Negotiation successful
0620 09:30:04.712861 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67,"mutex_wait_us":5}
I20260620 09:30:04.712965 3175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:04.710264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.129:34655 (local address 127.16.43.131:33297)
0620 09:30:04.710324 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:04.710326 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:04.710336 (+ 10us) server_negotiation.cc:407] Connection header received
0620 09:30:04.710371 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:04.710372 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:04.710384 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:04.710480 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:30:04.710718 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:04.711203 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:04.712234 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:04.712609 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:04.712666 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:04.712674 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:04.712843 (+ 169us) server_negotiation.cc:299] Negotiation successful
0620 09:30:04.712892 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 09:30:04.713042 3176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:04.710242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.129:39851 (local address 127.16.43.130:44489)
0620 09:30:04.710321 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:04.710324 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:04.710392 (+ 68us) server_negotiation.cc:407] Connection header received
0620 09:30:04.710455 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:04.710456 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:04.710463 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:04.710504 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:30:04.710761 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:04.711227 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:04.712287 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:04.712661 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:04.712740 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:04.712749 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:04.712928 (+ 179us) server_negotiation.cc:299] Negotiation successful
0620 09:30:04.712969 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 09:30:04.713106 3186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:04.710146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.130:44489 (local address 127.16.43.129:39851)
0620 09:30:04.710313 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 09:30:04.710324 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 09:30:04.710393 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:30:04.710519 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:30:04.710520 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:30:04.710522 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:30:04.710609 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:30:04.710611 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:30:04.711337 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:30:04.711338 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:30:04.712170 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:30:04.712173 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:30:04.712750 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:30:04.712751 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:30:04.712849 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:04.712853 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 09:30:04.712910 (+ 57us) client_negotiation.cc:240] Negotiation successful
0620 09:30:04.712947 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"mutex_wait_us":16,"thread_start_us":95,"threads_started":1}
I20260620 09:30:04.713356 3117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c4a7abb8dc44234b92f1b1f40f9f3e9" candidate_uuid: "2397eb6cc16a487484a50bf707518254" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d9426b8224c4426bd1fcb52ff033f13" is_pre_election: true
I20260620 09:30:04.713478 2986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c4a7abb8dc44234b92f1b1f40f9f3e9" candidate_uuid: "2397eb6cc16a487484a50bf707518254" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f78e6d1fa6c24f458715cda8b66dd2cf" is_pre_election: true
I20260620 09:30:04.713543 3117 raft_consensus.cc:2454] T 7c4a7abb8dc44234b92f1b1f40f9f3e9 P 7d9426b8224c4426bd1fcb52ff033f13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2397eb6cc16a487484a50bf707518254 in term 0.
I20260620 09:30:04.713641 2986 raft_consensus.cc:2454] T 7c4a7abb8dc44234b92f1b1f40f9f3e9 P f78e6d1fa6c24f458715cda8b66dd2cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2397eb6cc16a487484a50bf707518254 in term 0.
I20260620 09:30:04.713867 2789 leader_election.cc:302] T 7c4a7abb8dc44234b92f1b1f40f9f3e9 P 2397eb6cc16a487484a50bf707518254 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2397eb6cc16a487484a50bf707518254, f78e6d1fa6c24f458715cda8b66dd2cf; no voters:
I20260620 09:30:04.714056 3183 raft_consensus.cc:2781] T 7c4a7abb8dc44234b92f1b1f40f9f3e9 P 2397eb6cc16a487484a50bf707518254 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 09:30:04.714095 3183 raft_consensus.cc:482] T 7c4a7abb8dc44234b92f1b1f40f9f3e9 P 2397eb6cc16a487484a50bf707518254 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 09:30:04.714111 3183 raft_consensus.cc:3037] T 7c4a7abb8dc44234b92f1b1f40f9f3e9 P 2397eb6cc16a487484a50bf707518254 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:30:04.714882 3183 raft_consensus.cc:504] T 7c4a7abb8dc44234b92f1b1f40f9f3e9 P 2397eb6cc16a487484a50bf707518254 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2397eb6cc16a487484a50bf707518254" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 41595 } } peers { permanent_uuid: "7d9426b8224c4426bd1fcb52ff033f13" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 33297 } } peers { permanent_uuid: "f78e6d1fa6c24f458715cda8b66dd2cf" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 44489 } }
I20260620 09:30:04.715095 3183 leader_election.cc:288] T 7c4a7abb8dc44234b92f1b1f40f9f3e9 P 2397eb6cc16a487484a50bf707518254 [CANDIDATE]: Term 1 election: Requested vote from peers 7d9426b8224c4426bd1fcb52ff033f13 (127.16.43.131:33297), f78e6d1fa6c24f458715cda8b66dd2cf (127.16.43.130:44489)
I20260620 09:30:04.715197 2986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c4a7abb8dc44234b92f1b1f40f9f3e9" candidate_uuid: "2397eb6cc16a487484a50bf707518254" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f78e6d1fa6c24f458715cda8b66dd2cf"
I20260620 09:30:04.715268 2986 raft_consensus.cc:3037] T 7c4a7abb8dc44234b92f1b1f40f9f3e9 P f78e6d1fa6c24f458715cda8b66dd2cf [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:30:04.715268 3117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c4a7abb8dc44234b92f1b1f40f9f3e9" candidate_uuid: "2397eb6cc16a487484a50bf707518254" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d9426b8224c4426bd1fcb52ff033f13"
I20260620 09:30:04.715346 3117 raft_consensus.cc:3037] T 7c4a7abb8dc44234b92f1b1f40f9f3e9 P 7d9426b8224c4426bd1fcb52ff033f13 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:30:04.716010 2986 raft_consensus.cc:2454] T 7c4a7abb8dc44234b92f1b1f40f9f3e9 P f78e6d1fa6c24f458715cda8b66dd2cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2397eb6cc16a487484a50bf707518254 in term 1.
I20260620 09:30:04.716219 3117 raft_consensus.cc:2454] T 7c4a7abb8dc44234b92f1b1f40f9f3e9 P 7d9426b8224c4426bd1fcb52ff033f13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2397eb6cc16a487484a50bf707518254 in term 1.
I20260620 09:30:04.716223 2789 leader_election.cc:302] T 7c4a7abb8dc44234b92f1b1f40f9f3e9 P 2397eb6cc16a487484a50bf707518254 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2397eb6cc16a487484a50bf707518254, f78e6d1fa6c24f458715cda8b66dd2cf; no voters:
I20260620 09:30:04.716423 3183 raft_consensus.cc:2781] T 7c4a7abb8dc44234b92f1b1f40f9f3e9 P 2397eb6cc16a487484a50bf707518254 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:30:04.716588 3183 raft_consensus.cc:686] T 7c4a7abb8dc44234b92f1b1f40f9f3e9 P 2397eb6cc16a487484a50bf707518254 [term 1 LEADER]: Becoming Leader. State: Replica: 2397eb6cc16a487484a50bf707518254, State: Running, Role: LEADER
I20260620 09:30:04.716701 3183 consensus_queue.cc:237] T 7c4a7abb8dc44234b92f1b1f40f9f3e9 P 2397eb6cc16a487484a50bf707518254 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2397eb6cc16a487484a50bf707518254" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 41595 } } peers { permanent_uuid: "7d9426b8224c4426bd1fcb52ff033f13" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 33297 } } peers { permanent_uuid: "f78e6d1fa6c24f458715cda8b66dd2cf" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 44489 } }
I20260620 09:30:04.717672 2712 catalog_manager.cc:5495] T 7c4a7abb8dc44234b92f1b1f40f9f3e9 P 2397eb6cc16a487484a50bf707518254 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2397eb6cc16a487484a50bf707518254 (127.16.43.129). New cstate: current_term: 1 leader_uuid: "2397eb6cc16a487484a50bf707518254" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2397eb6cc16a487484a50bf707518254" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 41595 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7d9426b8224c4426bd1fcb52ff033f13" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 33297 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f78e6d1fa6c24f458715cda8b66dd2cf" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 44489 } health_report { overall_health: UNKNOWN } } }
I20260620 09:30:04.727216 3174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:04.724609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42432 (local address 127.16.43.129:41595)
0620 09:30:04.724688 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:04.724690 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:04.724699 (+ 9us) server_negotiation.cc:407] Connection header received
0620 09:30:04.724773 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:04.724774 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:04.724792 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:04.724860 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:30:04.724979 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:04.725503 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:04.726309 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:04.726498 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:04.726560 (+ 62us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:04.726774 (+ 214us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:30:04.726861 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:30:04.726938 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:04.726984 (+ 46us) server_negotiation.cc:299] Negotiation successful
0620 09:30:04.726994 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 09:30:04.729234 2986 raft_consensus.cc:1261] T 7c4a7abb8dc44234b92f1b1f40f9f3e9 P f78e6d1fa6c24f458715cda8b66dd2cf [term 1 FOLLOWER]: Refusing update from remote peer 2397eb6cc16a487484a50bf707518254: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:30:04.729244 3117 raft_consensus.cc:1261] T 7c4a7abb8dc44234b92f1b1f40f9f3e9 P 7d9426b8224c4426bd1fcb52ff033f13 [term 1 FOLLOWER]: Refusing update from remote peer 2397eb6cc16a487484a50bf707518254: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:30:04.729686 3183 consensus_queue.cc:1035] T 7c4a7abb8dc44234b92f1b1f40f9f3e9 P 2397eb6cc16a487484a50bf707518254 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f78e6d1fa6c24f458715cda8b66dd2cf" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 44489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:30:04.729837 3187 consensus_queue.cc:1035] T 7c4a7abb8dc44234b92f1b1f40f9f3e9 P 2397eb6cc16a487484a50bf707518254 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d9426b8224c4426bd1fcb52ff033f13" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 33297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:30:04.732688 3192 mvcc.cc:205] Tried to move back new op lower bound from 7298858208167940096 to 7298858208120315904. Current Snapshot: MvccSnapshot[applied={T|T < 7298858208167940096}]
I20260620 09:30:04.733093 3191 mvcc.cc:205] Tried to move back new op lower bound from 7298858208167940096 to 7298858208120315904. Current Snapshot: MvccSnapshot[applied={T|T < 7298858208167940096}]
I20260620 09:30:04.734752 2713 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52696:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:30:04.734858 2713 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52696:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:30:04.735651 2713 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e7708d1ca8fc4b10943c2a6ad9369dce: Sending DeleteTablet for 3 replicas of tablet 7c4a7abb8dc44234b92f1b1f40f9f3e9
I20260620 09:30:04.735976 2835 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c4a7abb8dc44234b92f1b1f40f9f3e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:30:04 UTC) from {username='slave'} at 127.0.0.1:42420
I20260620 09:30:04.736021 2966 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c4a7abb8dc44234b92f1b1f40f9f3e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:30:04 UTC) from {username='slave'} at 127.0.0.1:46948
I20260620 09:30:04.736145 3097 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c4a7abb8dc44234b92f1b1f40f9f3e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:30:04 UTC) from {username='slave'} at 127.0.0.1:34734
I20260620 09:30:04.736308 3201 tablet_replica.cc:330] T 7c4a7abb8dc44234b92f1b1f40f9f3e9 P f78e6d1fa6c24f458715cda8b66dd2cf: stopping tablet replica
I20260620 09:30:04.736418 3201 raft_consensus.cc:2229] T 7c4a7abb8dc44234b92f1b1f40f9f3e9 P f78e6d1fa6c24f458715cda8b66dd2cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:30:04.736531 3201 raft_consensus.cc:2258] T 7c4a7abb8dc44234b92f1b1f40f9f3e9 P f78e6d1fa6c24f458715cda8b66dd2cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:30:04.736637 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 2773
I20260620 09:30:04.737025 3201 ts_tablet_manager.cc:1905] T 7c4a7abb8dc44234b92f1b1f40f9f3e9 P f78e6d1fa6c24f458715cda8b66dd2cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:30:04.738369 3202 tablet_replica.cc:330] T 7c4a7abb8dc44234b92f1b1f40f9f3e9 P 7d9426b8224c4426bd1fcb52ff033f13: stopping tablet replica
I20260620 09:30:04.738466 3202 raft_consensus.cc:2229] T 7c4a7abb8dc44234b92f1b1f40f9f3e9 P 7d9426b8224c4426bd1fcb52ff033f13 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:30:04.738544 3202 raft_consensus.cc:2258] T 7c4a7abb8dc44234b92f1b1f40f9f3e9 P 7d9426b8224c4426bd1fcb52ff033f13 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:30:04.738710 3201 ts_tablet_manager.cc:1918] T 7c4a7abb8dc44234b92f1b1f40f9f3e9 P f78e6d1fa6c24f458715cda8b66dd2cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:30:04.738782 3201 log.cc:1196] T 7c4a7abb8dc44234b92f1b1f40f9f3e9 P f78e6d1fa6c24f458715cda8b66dd2cf: Deleting WAL directory at /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781947562949448-16558-0/minicluster-data/ts-1/wal/wals/7c4a7abb8dc44234b92f1b1f40f9f3e9
I20260620 09:30:04.739020 3201 ts_tablet_manager.cc:1939] T 7c4a7abb8dc44234b92f1b1f40f9f3e9 P f78e6d1fa6c24f458715cda8b66dd2cf: Deleting consensus metadata
I20260620 09:30:04.739176 3202 ts_tablet_manager.cc:1905] T 7c4a7abb8dc44234b92f1b1f40f9f3e9 P 7d9426b8224c4426bd1fcb52ff033f13: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:30:04.739472 2698 catalog_manager.cc:4841] TS f78e6d1fa6c24f458715cda8b66dd2cf (127.16.43.130:44489): tablet 7c4a7abb8dc44234b92f1b1f40f9f3e9 (table test-table [id=6f6689203415451e8e28a3094ec2abbe]) successfully deleted
I20260620 09:30:04.741183 3202 ts_tablet_manager.cc:1918] T 7c4a7abb8dc44234b92f1b1f40f9f3e9 P 7d9426b8224c4426bd1fcb52ff033f13: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:30:04.741263 3202 log.cc:1196] T 7c4a7abb8dc44234b92f1b1f40f9f3e9 P 7d9426b8224c4426bd1fcb52ff033f13: Deleting WAL directory at /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781947562949448-16558-0/minicluster-data/ts-2/wal/wals/7c4a7abb8dc44234b92f1b1f40f9f3e9
I20260620 09:30:04.741492 3202 ts_tablet_manager.cc:1939] T 7c4a7abb8dc44234b92f1b1f40f9f3e9 P 7d9426b8224c4426bd1fcb52ff033f13: Deleting consensus metadata
I20260620 09:30:04.742009 2699 catalog_manager.cc:4841] TS 7d9426b8224c4426bd1fcb52ff033f13 (127.16.43.131:33297): tablet 7c4a7abb8dc44234b92f1b1f40f9f3e9 (table test-table [id=6f6689203415451e8e28a3094ec2abbe]) successfully deleted
W20260620 09:30:04.744151 2700 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.43.129:41595 (error 108)
I20260620 09:30:04.744654 3172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:04.744511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.129:41595 (local address 127.0.0.1:42446)
0620 09:30:04.744573 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0620 09:30:04.744616 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.129:41595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260620 09:30:04.744756 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 2904
W20260620 09:30:04.744793 2700 catalog_manager.cc:4584] TS 2397eb6cc16a487484a50bf707518254 (127.16.43.129:41595): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7c4a7abb8dc44234b92f1b1f40f9f3e9: Network error: Client connection negotiation failed: client connection to 127.16.43.129:41595: connect: Connection refused (error 111)
I20260620 09:30:04.750200 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 3035
I20260620 09:30:04.754966 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 2682
2026-06-20T09:30:04Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (716 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781947562949448-16558-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:30:04 dist-test-slave-706q krb5kdc[3206](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:30:04 dist-test-slave-706q krb5kdc[3206](info): set up 2 sockets
Jun 20 09:30:04 dist-test-slave-706q krb5kdc[3206](info): commencing operation
krb5kdc: starting...
W20260620 09:30:05.808791 16558 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781947562949448-16558-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781947562949448-16558-0/krb5kdc/test-user.keytab.
Jun 20 09:30:05 dist-test-slave-706q krb5kdc[3206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947805, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:30:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:30:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.43.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.43.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.43.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.190.keytab.
Entry for principal kudu/127.16.43.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.190.keytab.
I20260620 09:30:05.942325 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781947562949448-16558-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781947562949448-16558-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781947562949448-16558-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.43.190:38615
--webserver_interface=127.16.43.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.190.keytab
--principal=kudu/127.16.43.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.43.148:40233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.43.190:38615
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781947562949448-16558-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781947562949448-16558-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781947562949448-16558-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:30:06.025942 3219 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:30:06.026160 3219 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:30:06.026185 3219 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:30:06.027390 3219 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:30:06.027432 3219 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:30:06.027447 3219 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:30:06.027459 3219 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:30:06.027472 3219 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:30:06.029186 3219 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:40233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781947562949448-16558-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781947562949448-16558-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.43.190:38615
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.190.keytab
--principal=kudu/127.16.43.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.43.190:38615
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.43.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3219
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781947562949448-16558-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:30:06.029445 3219 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:30:06.029722 3219 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:30:06.034734 3227 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:30:06.034757 3225 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:30:06.034858 3219 server_base.cc:1034] running on GCE node
W20260620 09:30:06.034811 3224 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:30:06.035177 3219 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:30:06.035493 3219 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:30:06.036728 3219 hybrid_clock.cc:648] HybridClock initialized: now 1781947806036734 us; error 77 us; skew 500 ppm
Jun 20 09:30:06 dist-test-slave-706q krb5kdc[3206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947806, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:30:06.039454 3219 init.cc:377] Logged in from keytab as kudu/127.16.43.190@KRBTEST.COM (short username kudu)
I20260620 09:30:06.039989 3219 webserver.cc:460] Webserver started at http://127.16.43.190:33445/ using document root <none> and password file <none>
I20260620 09:30:06.040217 3219 fs_manager.cc:359] Metadata directory not provided
I20260620 09:30:06.040287 3219 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:30:06.040443 3219 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:30:06.041592 3219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781947562949448-16558-0/minicluster-data/master-0/data/instance:
uuid: "c119f98246eb4308a054f02b32a59163"
format_stamp: "Formatted at 2026-06-20 09:30:06 on dist-test-slave-706q"
server_key: "86c004e90591ab8c8364464cd37b3ebc"
server_key_iv: "4f61efe7ab77ffb2d786df51eb1fc377"
server_key_version: "encryptionkey@0"
I20260620 09:30:06.041978 3219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781947562949448-16558-0/minicluster-data/master-0/wal/instance:
uuid: "c119f98246eb4308a054f02b32a59163"
format_stamp: "Formatted at 2026-06-20 09:30:06 on dist-test-slave-706q"
server_key: "86c004e90591ab8c8364464cd37b3ebc"
server_key_iv: "4f61efe7ab77ffb2d786df51eb1fc377"
server_key_version: "encryptionkey@0"
I20260620 09:30:06.043462 3219 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 09:30:06.044399 3234 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:30:06.044642 3219 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:30:06.044763 3219 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781947562949448-16558-0/minicluster-data/master-0/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781947562949448-16558-0/minicluster-data/master-0/wal
uuid: "c119f98246eb4308a054f02b32a59163"
format_stamp: "Formatted at 2026-06-20 09:30:06 on dist-test-slave-706q"
server_key: "86c004e90591ab8c8364464cd37b3ebc"
server_key_iv: "4f61efe7ab77ffb2d786df51eb1fc377"
server_key_version: "encryptionkey@0"
I20260620 09:30:06.044857 3219 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781947562949448-16558-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781947562949448-16558-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781947562949448-16558-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:30:06.065932 3219 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:30:06.068221 3219 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:30:06.068434 3219 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:30:06.072678 3219 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.190:38615
I20260620 09:30:06.072718 3286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.190:38615 every 8 connection(s)
I20260620 09:30:06.073073 3219 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
I20260620 09:30:06.073948 3287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:30:06.076340 3287 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c119f98246eb4308a054f02b32a59163: Bootstrap starting.
I20260620 09:30:06.076875 3287 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c119f98246eb4308a054f02b32a59163: Neither blocks nor log segments found. Creating new log.
I20260620 09:30:06.077114 3287 log.cc:824] T 00000000000000000000000000000000 P c119f98246eb4308a054f02b32a59163: Log is configured to *not* fsync() on all Append() calls
I20260620 09:30:06.077808 3287 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c119f98246eb4308a054f02b32a59163: No bootstrap required, opened a new log
I20260620 09:30:06.077972 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 3219
I20260620 09:30:06.078110 16558 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781947562949448-16558-0/minicluster-data/master-0/wal/instance
I20260620 09:30:06.078307 16558 external_mini_cluster.cc:1428] Setting key acea2ec32fbb81a6a94e6c66f9511496
I20260620 09:30:06.080621 3287 raft_consensus.cc:348] T 00000000000000000000000000000000 P c119f98246eb4308a054f02b32a59163 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c119f98246eb4308a054f02b32a59163" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 38615 } }
I20260620 09:30:06.080808 3287 raft_consensus.cc:374] T 00000000000000000000000000000000 P c119f98246eb4308a054f02b32a59163 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:30:06.080848 3287 raft_consensus.cc:729] T 00000000000000000000000000000000 P c119f98246eb4308a054f02b32a59163 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c119f98246eb4308a054f02b32a59163, State: Initialized, Role: FOLLOWER
I20260620 09:30:06.080986 3287 consensus_queue.cc:260] T 00000000000000000000000000000000 P c119f98246eb4308a054f02b32a59163 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c119f98246eb4308a054f02b32a59163" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 38615 } }
I20260620 09:30:06.081068 3287 raft_consensus.cc:388] T 00000000000000000000000000000000 P c119f98246eb4308a054f02b32a59163 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:30:06.081091 3287 raft_consensus.cc:482] T 00000000000000000000000000000000 P c119f98246eb4308a054f02b32a59163 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:30:06.081142 3287 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c119f98246eb4308a054f02b32a59163 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:30:06.081820 3287 raft_consensus.cc:504] T 00000000000000000000000000000000 P c119f98246eb4308a054f02b32a59163 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c119f98246eb4308a054f02b32a59163" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 38615 } }
I20260620 09:30:06.081909 3287 leader_election.cc:302] T 00000000000000000000000000000000 P c119f98246eb4308a054f02b32a59163 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c119f98246eb4308a054f02b32a59163; no voters:
I20260620 09:30:06.082125 3287 leader_election.cc:288] T 00000000000000000000000000000000 P c119f98246eb4308a054f02b32a59163 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:30:06.082156 3292 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c119f98246eb4308a054f02b32a59163 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:30:06.082262 3292 raft_consensus.cc:686] T 00000000000000000000000000000000 P c119f98246eb4308a054f02b32a59163 [term 1 LEADER]: Becoming Leader. State: Replica: c119f98246eb4308a054f02b32a59163, State: Running, Role: LEADER
I20260620 09:30:06.082367 3292 consensus_queue.cc:237] T 00000000000000000000000000000000 P c119f98246eb4308a054f02b32a59163 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c119f98246eb4308a054f02b32a59163" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 38615 } }
I20260620 09:30:06.082401 3287 sys_catalog.cc:564] T 00000000000000000000000000000000 P c119f98246eb4308a054f02b32a59163 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:30:06.082860 3293 sys_catalog.cc:455] T 00000000000000000000000000000000 P c119f98246eb4308a054f02b32a59163 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c119f98246eb4308a054f02b32a59163" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c119f98246eb4308a054f02b32a59163" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 38615 } } }
I20260620 09:30:06.082856 3294 sys_catalog.cc:455] T 00000000000000000000000000000000 P c119f98246eb4308a054f02b32a59163 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c119f98246eb4308a054f02b32a59163. Latest consensus state: current_term: 1 leader_uuid: "c119f98246eb4308a054f02b32a59163" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c119f98246eb4308a054f02b32a59163" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 38615 } } }
I20260620 09:30:06.082966 3293 sys_catalog.cc:458] T 00000000000000000000000000000000 P c119f98246eb4308a054f02b32a59163 [sys.catalog]: This master's current role is: LEADER
I20260620 09:30:06.082988 3294 sys_catalog.cc:458] T 00000000000000000000000000000000 P c119f98246eb4308a054f02b32a59163 [sys.catalog]: This master's current role is: LEADER
I20260620 09:30:06.083201 3298 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:30:06.084072 3298 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:30:06.085655 3298 catalog_manager.cc:1347] Generated new cluster ID: 5c58967a4853415d84294bf85c7b1d9a
I20260620 09:30:06.085708 3298 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 09:30:06 dist-test-slave-706q krb5kdc[3206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947805, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.43.190@KRBTEST.COM
I20260620 09:30:06.093962 3291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:06.078881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48436 (local address 127.16.43.190:38615)
0620 09:30:06.079098 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:06.079107 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:06.079142 (+ 35us) server_negotiation.cc:407] Connection header received
0620 09:30:06.080566 (+ 1424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:06.080575 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:06.080641 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:06.080816 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:30:06.081651 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:06.084855 (+ 3204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:06.085964 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:06.086574 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:06.086696 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:06.089589 (+ 2893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:30:06.089594 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:30:06.089594 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:30:06.089606 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:30:06.092264 (+ 2658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:30:06.092700 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:30:06.092703 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:30:06.092704 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:30:06.092807 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:30:06.093031 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:30:06.093032 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:30:06.093032 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:30:06.093324 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:30:06.093465 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:06.093592 (+ 127us) server_negotiation.cc:299] Negotiation successful
0620 09:30:06.093737 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":101,"threads_started":1}
I20260620 09:30:06.111258 3298 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:30:06.111897 3298 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:30:06.115903 3298 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c119f98246eb4308a054f02b32a59163: Generated new TSK 0
I20260620 09:30:06.116117 3298 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.43.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.43.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.43.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.129.keytab.
Entry for principal kudu/127.16.43.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.129.keytab.
I20260620 09:30:06.147796 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781947562949448-16558-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781947562949448-16558-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781947562949448-16558-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781947562949448-16558-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.43.129:0
--local_ip_for_outbound_sockets=127.16.43.129
--webserver_interface=127.16.43.129
--webserver_port=0
--tserver_master_addrs=127.16.43.190:38615
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.129.keytab
--principal=kudu/127.16.43.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.43.148:40233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781947562949448-16558-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781947562949448-16558-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781947562949448-16558-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:30:06.231213 3313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:30:06.231411 3313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:30:06.231436 3313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:30:06.232721 3313 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:30:06.232764 3313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:30:06.232791 3313 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.43.129
I20260620 09:30:06.234215 3313 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:40233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781947562949448-16558-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781947562949448-16558-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.129.keytab
--principal=kudu/127.16.43.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.43.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781947562949448-16558-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.43.129
--webserver_port=0
--tserver_master_addrs=127.16.43.190:38615
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3313
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.43.129
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781947562949448-16558-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:30:06.234405 3313 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:30:06.234594 3313 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:30:06.239948 3319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:30:06.239948 3318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:30:06.240142 3313 server_base.cc:1034] running on GCE node
W20260620 09:30:06.240011 3321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:30:06.240429 3313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:30:06.240685 3313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:30:06.241827 3313 hybrid_clock.cc:648] HybridClock initialized: now 1781947806241817 us; error 28 us; skew 500 ppm
Jun 20 09:30:06 dist-test-slave-706q krb5kdc[3206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947806, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:30:06.244485 3313 init.cc:377] Logged in from keytab as kudu/127.16.43.129@KRBTEST.COM (short username kudu)
I20260620 09:30:06.244920 3313 webserver.cc:460] Webserver started at http://127.16.43.129:38825/ using document root <none> and password file <none>
I20260620 09:30:06.245162 3313 fs_manager.cc:359] Metadata directory not provided
I20260620 09:30:06.245213 3313 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:30:06.245381 3313 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:30:06.246466 3313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781947562949448-16558-0/minicluster-data/ts-0/data/instance:
uuid: "4b96223312784ef3820d2e8b2278e232"
format_stamp: "Formatted at 2026-06-20 09:30:06 on dist-test-slave-706q"
server_key: "bb3602e31ed67fb78d4184c6c25531bd"
server_key_iv: "2d59b449f7665f5c15eb3316f19a8e43"
server_key_version: "encryptionkey@0"
I20260620 09:30:06.246824 3313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781947562949448-16558-0/minicluster-data/ts-0/wal/instance:
uuid: "4b96223312784ef3820d2e8b2278e232"
format_stamp: "Formatted at 2026-06-20 09:30:06 on dist-test-slave-706q"
server_key: "bb3602e31ed67fb78d4184c6c25531bd"
server_key_iv: "2d59b449f7665f5c15eb3316f19a8e43"
server_key_version: "encryptionkey@0"
I20260620 09:30:06.248306 3313 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 09:30:06.249202 3328 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:30:06.249527 3313 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 09:30:06.249601 3313 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781947562949448-16558-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781947562949448-16558-0/minicluster-data/ts-0/wal
uuid: "4b96223312784ef3820d2e8b2278e232"
format_stamp: "Formatted at 2026-06-20 09:30:06 on dist-test-slave-706q"
server_key: "bb3602e31ed67fb78d4184c6c25531bd"
server_key_iv: "2d59b449f7665f5c15eb3316f19a8e43"
server_key_version: "encryptionkey@0"
I20260620 09:30:06.249747 3313 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781947562949448-16558-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781947562949448-16558-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781947562949448-16558-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:30:06.266129 3313 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:30:06.268529 3313 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:30:06.268674 3313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:30:06.268940 3313 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:30:06.269279 3313 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:30:06.269315 3313 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:30:06.269338 3313 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:30:06.269351 3313 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:30:06.275462 3313 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.129:40815
I20260620 09:30:06.275558 3441 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.129:40815 every 8 connection(s)
I20260620 09:30:06.275868 3313 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781947562949448-16558-0/minicluster-data/ts-0/data/info.pb
Jun 20 09:30:06 dist-test-slave-706q krb5kdc[3206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947806, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.129@KRBTEST.COM for kudu/127.16.43.190@KRBTEST.COM
I20260620 09:30:06.283177 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 3313
I20260620 09:30:06.283320 16558 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781947562949448-16558-0/minicluster-data/ts-0/wal/instance
I20260620 09:30:06.283567 16558 external_mini_cluster.cc:1428] Setting key 911c28c934fc559da76baeece87f1b97
I20260620 09:30:06.287055 3291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:06.276675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.129:53055 (local address 127.16.43.190:38615)
0620 09:30:06.276726 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:06.276728 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:06.276973 (+ 245us) server_negotiation.cc:407] Connection header received
0620 09:30:06.277547 (+ 574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:06.277548 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:06.277561 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:06.277622 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:30:06.278872 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:06.279422 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:06.280489 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:06.280738 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:06.280830 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:06.283513 (+ 2683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:30:06.283516 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:30:06.283516 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:30:06.283547 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:30:06.285439 (+ 1892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:30:06.285968 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:30:06.285969 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:30:06.285970 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:30:06.286037 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:30:06.286335 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:30:06.286336 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:30:06.286336 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:30:06.286459 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:30:06.286539 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:06.286854 (+ 315us) server_negotiation.cc:299] Negotiation successful
0620 09:30:06.286951 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 09:30:06.287314 3444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:06.276585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:38615 (local address 127.16.43.129:53055)
0620 09:30:06.276844 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0620 09:30:06.276882 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 09:30:06.277433 (+ 551us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:30:06.277703 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:30:06.277713 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:30:06.278292 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:30:06.278740 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:30:06.278748 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:30:06.279575 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:30:06.279576 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:30:06.280381 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:30:06.280383 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:30:06.280845 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:30:06.280845 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:30:06.280975 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:06.281561 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:30:06.281619 (+ 58us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:30:06.283315 (+ 1696us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:30:06.285546 (+ 2231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:30:06.285553 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:30:06.285555 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:30:06.285878 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:30:06.286112 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:30:06.286113 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:30:06.286113 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:30:06.286253 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:30:06.286530 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:30:06.286538 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:30:06.286702 (+ 164us) client_negotiation.cc:769] Sending connection context
0620 09:30:06.286833 (+ 131us) client_negotiation.cc:240] Negotiation successful
0620 09:30:06.287011 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":149,"threads_started":1}
I20260620 09:30:06.287829 3442 heartbeater.cc:347] Connected to a master server at 127.16.43.190:38615
I20260620 09:30:06.287945 3442 heartbeater.cc:464] Registering TS with master...
I20260620 09:30:06.288180 3442 heartbeater.cc:511] Master 127.16.43.190:38615 requested a full tablet report, sending...
I20260620 09:30:06.288766 3251 ts_manager.cc:194] Registered new tserver with Master: 4b96223312784ef3820d2e8b2278e232 (127.16.43.129:40815)
I20260620 09:30:06.289503 3251 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.43.129@KRBTEST.COM'} at 127.16.43.129:53055
WARNING: no policy specified for kudu/127.16.43.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.43.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.43.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.130.keytab.
Entry for principal kudu/127.16.43.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.130.keytab.
I20260620 09:30:06.311401 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781947562949448-16558-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781947562949448-16558-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781947562949448-16558-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781947562949448-16558-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.43.130:0
--local_ip_for_outbound_sockets=127.16.43.130
--webserver_interface=127.16.43.130
--webserver_port=0
--tserver_master_addrs=127.16.43.190:38615
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.130.keytab
--principal=kudu/127.16.43.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.43.148:40233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781947562949448-16558-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781947562949448-16558-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781947562949448-16558-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:30:06.395062 3447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:30:06.395362 3447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:30:06.395406 3447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:30:06.397154 3447 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:30:06.397213 3447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:30:06.397250 3447 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.43.130
I20260620 09:30:06.399327 3447 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:40233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781947562949448-16558-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781947562949448-16558-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.130.keytab
--principal=kudu/127.16.43.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.43.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781947562949448-16558-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.43.130
--webserver_port=0
--tserver_master_addrs=127.16.43.190:38615
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3447
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.43.130
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781947562949448-16558-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:30:06.399583 3447 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:30:06.399847 3447 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:30:06.406162 3452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:30:06.406163 3453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:30:06.406507 3447 server_base.cc:1034] running on GCE node
W20260620 09:30:06.406733 3455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:30:06.406984 3447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:30:06.407275 3447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:30:06.408453 3447 hybrid_clock.cc:648] HybridClock initialized: now 1781947806408423 us; error 48 us; skew 500 ppm
Jun 20 09:30:06 dist-test-slave-706q krb5kdc[3206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947806, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:30:06.411068 3447 init.cc:377] Logged in from keytab as kudu/127.16.43.130@KRBTEST.COM (short username kudu)
I20260620 09:30:06.411559 3447 webserver.cc:460] Webserver started at http://127.16.43.130:38193/ using document root <none> and password file <none>
I20260620 09:30:06.411821 3447 fs_manager.cc:359] Metadata directory not provided
I20260620 09:30:06.411911 3447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:30:06.412048 3447 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:30:06.413182 3447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781947562949448-16558-0/minicluster-data/ts-1/data/instance:
uuid: "9d737dcf489d4fbfa03b962aaeeb557d"
format_stamp: "Formatted at 2026-06-20 09:30:06 on dist-test-slave-706q"
server_key: "1046a308cbe0baaea94672320d99d84b"
server_key_iv: "d98974b13c7838b70cbe928c43e715fc"
server_key_version: "encryptionkey@0"
I20260620 09:30:06.413548 3447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781947562949448-16558-0/minicluster-data/ts-1/wal/instance:
uuid: "9d737dcf489d4fbfa03b962aaeeb557d"
format_stamp: "Formatted at 2026-06-20 09:30:06 on dist-test-slave-706q"
server_key: "1046a308cbe0baaea94672320d99d84b"
server_key_iv: "d98974b13c7838b70cbe928c43e715fc"
server_key_version: "encryptionkey@0"
I20260620 09:30:06.414912 3447 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 09:30:06.415721 3462 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:30:06.415935 3447 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:30:06.416028 3447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781947562949448-16558-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781947562949448-16558-0/minicluster-data/ts-1/wal
uuid: "9d737dcf489d4fbfa03b962aaeeb557d"
format_stamp: "Formatted at 2026-06-20 09:30:06 on dist-test-slave-706q"
server_key: "1046a308cbe0baaea94672320d99d84b"
server_key_iv: "d98974b13c7838b70cbe928c43e715fc"
server_key_version: "encryptionkey@0"
I20260620 09:30:06.416109 3447 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781947562949448-16558-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781947562949448-16558-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781947562949448-16558-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:30:06.424718 3447 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:30:06.426678 3447 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:30:06.426836 3447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:30:06.427053 3447 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:30:06.427348 3447 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:30:06.427399 3447 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:30:06.427441 3447 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:30:06.427469 3447 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:30:06.433323 3447 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.130:33289
I20260620 09:30:06.433410 3575 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.130:33289 every 8 connection(s)
I20260620 09:30:06.433719 3447 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781947562949448-16558-0/minicluster-data/ts-1/data/info.pb
I20260620 09:30:06.436321 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 3447
I20260620 09:30:06.436442 16558 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781947562949448-16558-0/minicluster-data/ts-1/wal/instance
I20260620 09:30:06.436632 16558 external_mini_cluster.cc:1428] Setting key 3a6c8922e1ca9084836c581827b3f261
Jun 20 09:30:06 dist-test-slave-706q krb5kdc[3206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947806, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.130@KRBTEST.COM for kudu/127.16.43.190@KRBTEST.COM
I20260620 09:30:06.444435 3291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:06.434462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.130:37325 (local address 127.16.43.190:38615)
0620 09:30:06.434518 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:06.434521 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:06.434709 (+ 188us) server_negotiation.cc:407] Connection header received
0620 09:30:06.435168 (+ 459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:06.435168 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:06.435178 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:06.435263 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:30:06.436334 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:06.436981 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:06.438104 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:06.438434 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:06.438514 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:06.441242 (+ 2728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:30:06.441244 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:30:06.441244 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:30:06.441253 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:30:06.442961 (+ 1708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:30:06.443375 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:30:06.443377 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:30:06.443377 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:30:06.443443 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:30:06.443738 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:30:06.443740 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:30:06.443740 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:30:06.443897 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:30:06.443984 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:06.444231 (+ 247us) server_negotiation.cc:299] Negotiation successful
0620 09:30:06.444312 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 09:30:06.444470 3578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:06.434420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:38615 (local address 127.16.43.130:37325)
0620 09:30:06.434575 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 09:30:06.434617 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 09:30:06.435091 (+ 474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:30:06.435315 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:30:06.435319 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:30:06.435960 (+ 641us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:30:06.436207 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:30:06.436214 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:30:06.437079 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:30:06.437080 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:30:06.438014 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:30:06.438016 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:30:06.438539 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:30:06.438540 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:30:06.438713 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:06.439299 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:30:06.439307 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:30:06.441126 (+ 1819us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:30:06.443051 (+ 1925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:30:06.443057 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:30:06.443059 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:30:06.443300 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:30:06.443510 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:30:06.443511 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:30:06.443511 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:30:06.443670 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:30:06.443965 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:30:06.443970 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:30:06.444090 (+ 120us) client_negotiation.cc:769] Sending connection context
0620 09:30:06.444180 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 09:30:06.444304 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":84,"threads_started":1}
I20260620 09:30:06.445009 3576 heartbeater.cc:347] Connected to a master server at 127.16.43.190:38615
I20260620 09:30:06.445127 3576 heartbeater.cc:464] Registering TS with master...
I20260620 09:30:06.445374 3576 heartbeater.cc:511] Master 127.16.43.190:38615 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.16.43.131@KRBTEST.COM; defaulting to no policy
I20260620 09:30:06.445853 3251 ts_manager.cc:194] Registered new tserver with Master: 9d737dcf489d4fbfa03b962aaeeb557d (127.16.43.130:33289)
I20260620 09:30:06.446238 3251 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.43.130@KRBTEST.COM'} at 127.16.43.130:37325
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.43.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.43.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.131.keytab.
Entry for principal kudu/127.16.43.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.131.keytab.
I20260620 09:30:06.466727 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781947562949448-16558-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781947562949448-16558-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781947562949448-16558-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781947562949448-16558-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.43.131:0
--local_ip_for_outbound_sockets=127.16.43.131
--webserver_interface=127.16.43.131
--webserver_port=0
--tserver_master_addrs=127.16.43.190:38615
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.131.keytab
--principal=kudu/127.16.43.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.43.148:40233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781947562949448-16558-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781947562949448-16558-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781947562949448-16558-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:30:06.550783 3581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:30:06.551024 3581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:30:06.551065 3581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:30:06.552307 3581 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:30:06.552373 3581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:30:06.552418 3581 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.43.131
I20260620 09:30:06.553957 3581 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:40233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781947562949448-16558-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781947562949448-16558-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.131.keytab
--principal=kudu/127.16.43.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.43.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781947562949448-16558-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.43.131
--webserver_port=0
--tserver_master_addrs=127.16.43.190:38615
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3581
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.43.131
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781947562949448-16558-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:30:06.554234 3581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:30:06.554471 3581 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:30:06.559581 3586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:30:06.559630 3587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:30:06.559562 3589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:30:06.559851 3581 server_base.cc:1034] running on GCE node
I20260620 09:30:06.560027 3581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:30:06.560253 3581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:30:06.561399 3581 hybrid_clock.cc:648] HybridClock initialized: now 1781947806561383 us; error 28 us; skew 500 ppm
Jun 20 09:30:06 dist-test-slave-706q krb5kdc[3206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947806, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:30:06.563934 3581 init.cc:377] Logged in from keytab as kudu/127.16.43.131@KRBTEST.COM (short username kudu)
I20260620 09:30:06.564568 3581 webserver.cc:460] Webserver started at http://127.16.43.131:40049/ using document root <none> and password file <none>
I20260620 09:30:06.564857 3581 fs_manager.cc:359] Metadata directory not provided
I20260620 09:30:06.564917 3581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:30:06.565061 3581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:30:06.566720 3581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781947562949448-16558-0/minicluster-data/ts-2/data/instance:
uuid: "60073351d4c243a88b215a4e5fbde0f8"
format_stamp: "Formatted at 2026-06-20 09:30:06 on dist-test-slave-706q"
server_key: "c8d90d081a0ef90b59cbe793731f68f3"
server_key_iv: "0e3945163ef9421bbd5948f482af9156"
server_key_version: "encryptionkey@0"
I20260620 09:30:06.567159 3581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781947562949448-16558-0/minicluster-data/ts-2/wal/instance:
uuid: "60073351d4c243a88b215a4e5fbde0f8"
format_stamp: "Formatted at 2026-06-20 09:30:06 on dist-test-slave-706q"
server_key: "c8d90d081a0ef90b59cbe793731f68f3"
server_key_iv: "0e3945163ef9421bbd5948f482af9156"
server_key_version: "encryptionkey@0"
I20260620 09:30:06.568756 3581 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.004s
I20260620 09:30:06.575919 3596 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:30:06.576143 3581 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.001s sys 0.000s
I20260620 09:30:06.576251 3581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781947562949448-16558-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781947562949448-16558-0/minicluster-data/ts-2/wal
uuid: "60073351d4c243a88b215a4e5fbde0f8"
format_stamp: "Formatted at 2026-06-20 09:30:06 on dist-test-slave-706q"
server_key: "c8d90d081a0ef90b59cbe793731f68f3"
server_key_iv: "0e3945163ef9421bbd5948f482af9156"
server_key_version: "encryptionkey@0"
I20260620 09:30:06.576326 3581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781947562949448-16558-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781947562949448-16558-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781947562949448-16558-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:30:06.591188 3581 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:30:06.593705 3581 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:30:06.593827 3581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:30:06.594101 3581 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:30:06.594470 3581 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:30:06.594506 3581 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:30:06.594528 3581 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:30:06.594542 3581 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:30:06.600564 3581 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.131:34495
I20260620 09:30:06.600641 3709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.131:34495 every 8 connection(s)
I20260620 09:30:06.600939 3581 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781947562949448-16558-0/minicluster-data/ts-2/data/info.pb
I20260620 09:30:06.602159 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 3581
I20260620 09:30:06.602252 16558 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781947562949448-16558-0/minicluster-data/ts-2/wal/instance
I20260620 09:30:06.602425 16558 external_mini_cluster.cc:1428] Setting key e2f327223024d32173e1cdb9593542d9
Jun 20 09:30:06 dist-test-slave-706q krb5kdc[3206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947806, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.131@KRBTEST.COM for kudu/127.16.43.190@KRBTEST.COM
I20260620 09:30:06.611065 3291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:06.601779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.131:57825 (local address 127.16.43.190:38615)
0620 09:30:06.601871 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:06.601874 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:06.602063 (+ 189us) server_negotiation.cc:407] Connection header received
0620 09:30:06.602732 (+ 669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:06.602733 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:06.602747 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:06.602811 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:30:06.603924 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:06.604425 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:06.605428 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:06.605689 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:06.605774 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:06.608065 (+ 2291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:30:06.608066 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:30:06.608066 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:30:06.608074 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:30:06.609552 (+ 1478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:30:06.610119 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:30:06.610120 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:30:06.610121 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:30:06.610196 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:30:06.610455 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:30:06.610455 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:30:06.610456 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:30:06.610565 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:30:06.610671 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:06.610934 (+ 263us) server_negotiation.cc:299] Negotiation successful
0620 09:30:06.610997 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 09:30:06.611249 3712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:06.601712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:38615 (local address 127.16.43.131:57825)
0620 09:30:06.601873 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 09:30:06.601927 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 09:30:06.602613 (+ 686us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:30:06.602884 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:30:06.602888 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:30:06.603642 (+ 754us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:30:06.603840 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:30:06.603845 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:30:06.604524 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:30:06.604525 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:30:06.605334 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:30:06.605337 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:30:06.605814 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:30:06.605815 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:30:06.605930 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:06.606427 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:30:06.606435 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:30:06.607972 (+ 1537us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:30:06.609789 (+ 1817us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:30:06.609797 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:30:06.609800 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:30:06.610029 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:30:06.610283 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:30:06.610284 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:30:06.610285 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:30:06.610395 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:30:06.610668 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:30:06.610672 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:30:06.610823 (+ 151us) client_negotiation.cc:769] Sending connection context
0620 09:30:06.610910 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 09:30:06.611038 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":66,"threads_started":1}
I20260620 09:30:06.611743 3710 heartbeater.cc:347] Connected to a master server at 127.16.43.190:38615
I20260620 09:30:06.611837 3710 heartbeater.cc:464] Registering TS with master...
I20260620 09:30:06.611984 3710 heartbeater.cc:511] Master 127.16.43.190:38615 requested a full tablet report, sending...
I20260620 09:30:06.612380 3251 ts_manager.cc:194] Registered new tserver with Master: 60073351d4c243a88b215a4e5fbde0f8 (127.16.43.131:34495)
I20260620 09:30:06.612774 3251 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.43.131@KRBTEST.COM'} at 127.16.43.131:57825
I20260620 09:30:06.614073 16558 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 09:30:06.621645 3291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:06.615403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48950 (local address 127.16.43.190:38615)
0620 09:30:06.615458 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:06.615460 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:06.615581 (+ 121us) server_negotiation.cc:407] Connection header received
0620 09:30:06.615678 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:06.615679 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:06.615688 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:06.615744 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:30:06.616409 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:06.616902 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:06.617627 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:06.617831 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:06.617914 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:06.619097 (+ 1183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:30:06.619098 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:30:06.619098 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:30:06.619105 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:30:06.620441 (+ 1336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:30:06.620817 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:30:06.620819 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:30:06.620819 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:30:06.620875 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:30:06.621067 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:30:06.621068 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:30:06.621068 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:30:06.621227 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:30:06.621326 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:06.621450 (+ 124us) server_negotiation.cc:299] Negotiation successful
0620 09:30:06.621538 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 09:30:06.623214 3251 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48950:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 09:30:06.625025 3251 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 09:30:06.632215 3723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:06.627544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.131:34495 (local address 127.0.0.1:43254)
0620 09:30:06.628404 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0620 09:30:06.628413 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 09:30:06.628456 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:30:06.628727 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:30:06.628727 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:30:06.628729 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:30:06.628769 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:30:06.628770 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:30:06.629627 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:30:06.629628 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:30:06.630742 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:30:06.630745 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:30:06.631839 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:30:06.631840 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:30:06.631934 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:06.631950 (+ 16us) client_negotiation.cc:769] Sending connection context
0620 09:30:06.632026 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 09:30:06.632097 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":843,"thread_start_us":97,"threads_started":1}
I20260620 09:30:06.632220 3726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:06.627787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43254 (local address 127.16.43.131:34495)
0620 09:30:06.627982 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:06.627986 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:06.628515 (+ 529us) server_negotiation.cc:407] Connection header received
0620 09:30:06.628565 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:06.628567 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:06.628583 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:06.628649 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:30:06.628842 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:06.629524 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:06.630838 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:06.631727 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:06.631813 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:06.631881 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:06.632063 (+ 182us) server_negotiation.cc:299] Negotiation successful
0620 09:30:06.632113 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":88,"threads_started":1}
I20260620 09:30:06.632311 3725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:06.627790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37946 (local address 127.16.43.130:33289)
0620 09:30:06.627966 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:06.627970 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:06.628294 (+ 324us) server_negotiation.cc:407] Connection header received
0620 09:30:06.628488 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:06.628491 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:06.628507 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:06.628596 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:30:06.628829 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:06.630016 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:06.631181 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:06.631771 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:06.631858 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:06.631919 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:06.632181 (+ 262us) server_negotiation.cc:299] Negotiation successful
0620 09:30:06.632225 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":73,"threads_started":1}
I20260620 09:30:06.632709 3722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:06.627523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.130:33289 (local address 127.0.0.1:37946)
0620 09:30:06.628203 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0620 09:30:06.628212 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 09:30:06.628309 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:30:06.628682 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:30:06.628683 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:30:06.628684 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:30:06.628740 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:30:06.628741 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:30:06.630158 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:30:06.630159 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:30:06.631098 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:30:06.631100 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:30:06.631951 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:30:06.631951 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:30:06.632013 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:06.632017 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 09:30:06.632062 (+ 45us) client_negotiation.cc:240] Negotiation successful
0620 09:30:06.632098 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":662,"thread_start_us":159,"threads_started":1}
I20260620 09:30:06.632843 3724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:06.627749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40662 (local address 127.16.43.129:40815)
0620 09:30:06.627950 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:06.627955 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:06.627968 (+ 13us) server_negotiation.cc:407] Connection header received
0620 09:30:06.628009 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:06.628011 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:06.628026 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:06.628118 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:30:06.628442 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:06.629899 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:06.631507 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:06.632342 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:06.632429 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:06.632485 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:06.632691 (+ 206us) server_negotiation.cc:299] Negotiation successful
0620 09:30:06.632743 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":80,"threads_started":1}
I20260620 09:30:06.633191 3721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:06.627521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.129:40815 (local address 127.0.0.1:40662)
0620 09:30:06.627774 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0620 09:30:06.627815 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 09:30:06.627884 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:30:06.628175 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:30:06.628178 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:30:06.628181 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:30:06.628243 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:30:06.628245 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:30:06.630049 (+ 1804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:30:06.630051 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:30:06.631355 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:30:06.631358 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:30:06.632530 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:30:06.632532 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:30:06.632595 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:06.632599 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 09:30:06.632653 (+ 54us) client_negotiation.cc:240] Negotiation successful
0620 09:30:06.632687 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"mutex_wait_us":33,"thread_start_us":100,"threads_started":1}
I20260620 09:30:06.633575 3644 tablet_service.cc:1467] Processing CreateTablet for tablet 5147ef4a05de4a39a3db1fd89e775a54 (DEFAULT_TABLE table=test-table [id=2277900927e24585970d79ec0424297f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:30:06.633644 3510 tablet_service.cc:1467] Processing CreateTablet for tablet 5147ef4a05de4a39a3db1fd89e775a54 (DEFAULT_TABLE table=test-table [id=2277900927e24585970d79ec0424297f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:30:06.633952 3510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5147ef4a05de4a39a3db1fd89e775a54. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:30:06.633967 3644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5147ef4a05de4a39a3db1fd89e775a54. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:30:06.636308 3376 tablet_service.cc:1467] Processing CreateTablet for tablet 5147ef4a05de4a39a3db1fd89e775a54 (DEFAULT_TABLE table=test-table [id=2277900927e24585970d79ec0424297f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:30:06.636723 3376 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5147ef4a05de4a39a3db1fd89e775a54. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:30:06.636919 3727 tablet_bootstrap.cc:492] T 5147ef4a05de4a39a3db1fd89e775a54 P 9d737dcf489d4fbfa03b962aaeeb557d: Bootstrap starting.
I20260620 09:30:06.637580 3728 tablet_bootstrap.cc:492] T 5147ef4a05de4a39a3db1fd89e775a54 P 60073351d4c243a88b215a4e5fbde0f8: Bootstrap starting.
I20260620 09:30:06.637723 3727 tablet_bootstrap.cc:654] T 5147ef4a05de4a39a3db1fd89e775a54 P 9d737dcf489d4fbfa03b962aaeeb557d: Neither blocks nor log segments found. Creating new log.
I20260620 09:30:06.638279 3728 tablet_bootstrap.cc:654] T 5147ef4a05de4a39a3db1fd89e775a54 P 60073351d4c243a88b215a4e5fbde0f8: Neither blocks nor log segments found. Creating new log.
I20260620 09:30:06.638391 3727 log.cc:824] T 5147ef4a05de4a39a3db1fd89e775a54 P 9d737dcf489d4fbfa03b962aaeeb557d: Log is configured to *not* fsync() on all Append() calls
I20260620 09:30:06.638840 3728 log.cc:824] T 5147ef4a05de4a39a3db1fd89e775a54 P 60073351d4c243a88b215a4e5fbde0f8: Log is configured to *not* fsync() on all Append() calls
I20260620 09:30:06.639276 3727 tablet_bootstrap.cc:492] T 5147ef4a05de4a39a3db1fd89e775a54 P 9d737dcf489d4fbfa03b962aaeeb557d: No bootstrap required, opened a new log
I20260620 09:30:06.639369 3727 ts_tablet_manager.cc:1397] T 5147ef4a05de4a39a3db1fd89e775a54 P 9d737dcf489d4fbfa03b962aaeeb557d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 09:30:06.639679 3728 tablet_bootstrap.cc:492] T 5147ef4a05de4a39a3db1fd89e775a54 P 60073351d4c243a88b215a4e5fbde0f8: No bootstrap required, opened a new log
I20260620 09:30:06.639748 3728 ts_tablet_manager.cc:1397] T 5147ef4a05de4a39a3db1fd89e775a54 P 60073351d4c243a88b215a4e5fbde0f8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 09:30:06.640406 3731 tablet_bootstrap.cc:492] T 5147ef4a05de4a39a3db1fd89e775a54 P 4b96223312784ef3820d2e8b2278e232: Bootstrap starting.
I20260620 09:30:06.641168 3731 tablet_bootstrap.cc:654] T 5147ef4a05de4a39a3db1fd89e775a54 P 4b96223312784ef3820d2e8b2278e232: Neither blocks nor log segments found. Creating new log.
I20260620 09:30:06.641492 3727 raft_consensus.cc:348] T 5147ef4a05de4a39a3db1fd89e775a54 P 9d737dcf489d4fbfa03b962aaeeb557d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d737dcf489d4fbfa03b962aaeeb557d" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 33289 } } peers { permanent_uuid: "60073351d4c243a88b215a4e5fbde0f8" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 34495 } } peers { permanent_uuid: "4b96223312784ef3820d2e8b2278e232" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 40815 } }
I20260620 09:30:06.641685 3727 raft_consensus.cc:374] T 5147ef4a05de4a39a3db1fd89e775a54 P 9d737dcf489d4fbfa03b962aaeeb557d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:30:06.641721 3727 raft_consensus.cc:729] T 5147ef4a05de4a39a3db1fd89e775a54 P 9d737dcf489d4fbfa03b962aaeeb557d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d737dcf489d4fbfa03b962aaeeb557d, State: Initialized, Role: FOLLOWER
I20260620 09:30:06.641762 3731 log.cc:824] T 5147ef4a05de4a39a3db1fd89e775a54 P 4b96223312784ef3820d2e8b2278e232: Log is configured to *not* fsync() on all Append() calls
I20260620 09:30:06.641868 3727 consensus_queue.cc:260] T 5147ef4a05de4a39a3db1fd89e775a54 P 9d737dcf489d4fbfa03b962aaeeb557d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d737dcf489d4fbfa03b962aaeeb557d" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 33289 } } peers { permanent_uuid: "60073351d4c243a88b215a4e5fbde0f8" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 34495 } } peers { permanent_uuid: "4b96223312784ef3820d2e8b2278e232" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 40815 } }
I20260620 09:30:06.641858 3728 raft_consensus.cc:348] T 5147ef4a05de4a39a3db1fd89e775a54 P 60073351d4c243a88b215a4e5fbde0f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d737dcf489d4fbfa03b962aaeeb557d" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 33289 } } peers { permanent_uuid: "60073351d4c243a88b215a4e5fbde0f8" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 34495 } } peers { permanent_uuid: "4b96223312784ef3820d2e8b2278e232" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 40815 } }
I20260620 09:30:06.642027 3728 raft_consensus.cc:374] T 5147ef4a05de4a39a3db1fd89e775a54 P 60073351d4c243a88b215a4e5fbde0f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:30:06.642083 3728 raft_consensus.cc:729] T 5147ef4a05de4a39a3db1fd89e775a54 P 60073351d4c243a88b215a4e5fbde0f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60073351d4c243a88b215a4e5fbde0f8, State: Initialized, Role: FOLLOWER
I20260620 09:30:06.642207 3727 ts_tablet_manager.cc:1428] T 5147ef4a05de4a39a3db1fd89e775a54 P 9d737dcf489d4fbfa03b962aaeeb557d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 09:30:06.642202 3728 consensus_queue.cc:260] T 5147ef4a05de4a39a3db1fd89e775a54 P 60073351d4c243a88b215a4e5fbde0f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d737dcf489d4fbfa03b962aaeeb557d" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 33289 } } peers { permanent_uuid: "60073351d4c243a88b215a4e5fbde0f8" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 34495 } } peers { permanent_uuid: "4b96223312784ef3820d2e8b2278e232" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 40815 } }
I20260620 09:30:06.642338 3576 heartbeater.cc:503] Master 127.16.43.190:38615 was elected leader, sending a full tablet report...
I20260620 09:30:06.642478 3728 ts_tablet_manager.cc:1428] T 5147ef4a05de4a39a3db1fd89e775a54 P 60073351d4c243a88b215a4e5fbde0f8: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 09:30:06.642494 3710 heartbeater.cc:503] Master 127.16.43.190:38615 was elected leader, sending a full tablet report...
I20260620 09:30:06.643082 3731 tablet_bootstrap.cc:492] T 5147ef4a05de4a39a3db1fd89e775a54 P 4b96223312784ef3820d2e8b2278e232: No bootstrap required, opened a new log
I20260620 09:30:06.643165 3731 ts_tablet_manager.cc:1397] T 5147ef4a05de4a39a3db1fd89e775a54 P 4b96223312784ef3820d2e8b2278e232: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 09:30:06.645180 3731 raft_consensus.cc:348] T 5147ef4a05de4a39a3db1fd89e775a54 P 4b96223312784ef3820d2e8b2278e232 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d737dcf489d4fbfa03b962aaeeb557d" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 33289 } } peers { permanent_uuid: "60073351d4c243a88b215a4e5fbde0f8" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 34495 } } peers { permanent_uuid: "4b96223312784ef3820d2e8b2278e232" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 40815 } }
I20260620 09:30:06.645437 3731 raft_consensus.cc:374] T 5147ef4a05de4a39a3db1fd89e775a54 P 4b96223312784ef3820d2e8b2278e232 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:30:06.645498 3731 raft_consensus.cc:729] T 5147ef4a05de4a39a3db1fd89e775a54 P 4b96223312784ef3820d2e8b2278e232 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b96223312784ef3820d2e8b2278e232, State: Initialized, Role: FOLLOWER
I20260620 09:30:06.645632 3731 consensus_queue.cc:260] T 5147ef4a05de4a39a3db1fd89e775a54 P 4b96223312784ef3820d2e8b2278e232 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d737dcf489d4fbfa03b962aaeeb557d" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 33289 } } peers { permanent_uuid: "60073351d4c243a88b215a4e5fbde0f8" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 34495 } } peers { permanent_uuid: "4b96223312784ef3820d2e8b2278e232" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 40815 } }
I20260620 09:30:06.645941 3731 ts_tablet_manager.cc:1428] T 5147ef4a05de4a39a3db1fd89e775a54 P 4b96223312784ef3820d2e8b2278e232: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 09:30:06.646010 3442 heartbeater.cc:503] Master 127.16.43.190:38615 was elected leader, sending a full tablet report...
W20260620 09:30:06.684286 3577 tablet.cc:2369] T 5147ef4a05de4a39a3db1fd89e775a54 P 9d737dcf489d4fbfa03b962aaeeb557d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 09:30:06.728207 3735 raft_consensus.cc:482] T 5147ef4a05de4a39a3db1fd89e775a54 P 4b96223312784ef3820d2e8b2278e232 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:30:06.728364 3735 raft_consensus.cc:504] T 5147ef4a05de4a39a3db1fd89e775a54 P 4b96223312784ef3820d2e8b2278e232 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d737dcf489d4fbfa03b962aaeeb557d" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 33289 } } peers { permanent_uuid: "60073351d4c243a88b215a4e5fbde0f8" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 34495 } } peers { permanent_uuid: "4b96223312784ef3820d2e8b2278e232" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 40815 } }
I20260620 09:30:06.728729 3735 leader_election.cc:288] T 5147ef4a05de4a39a3db1fd89e775a54 P 4b96223312784ef3820d2e8b2278e232 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9d737dcf489d4fbfa03b962aaeeb557d (127.16.43.130:33289), 60073351d4c243a88b215a4e5fbde0f8 (127.16.43.131:34495)
I20260620 09:30:06.732988 3725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:06.729022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.129:39117 (local address 127.16.43.130:33289)
0620 09:30:06.729075 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:06.729077 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:06.729332 (+ 255us) server_negotiation.cc:407] Connection header received
0620 09:30:06.729366 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:06.729367 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:06.729377 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:06.729427 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:30:06.729588 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:06.730302 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:06.731973 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:06.732583 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:06.732667 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:06.732676 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:06.732867 (+ 191us) server_negotiation.cc:299] Negotiation successful
0620 09:30:06.732914 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 09:30:06.732988 3726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:06.729021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.129:53505 (local address 127.16.43.131:34495)
0620 09:30:06.729074 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:06.729076 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:06.729097 (+ 21us) server_negotiation.cc:407] Connection header received
0620 09:30:06.729196 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:06.729198 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:06.729241 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:06.729292 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:30:06.729490 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:06.730257 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:06.731973 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:06.732583 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:06.732666 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:06.732676 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:06.732877 (+ 201us) server_negotiation.cc:299] Negotiation successful
0620 09:30:06.732919 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 09:30:06.732993 3444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:06.728940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.131:34495 (local address 127.16.43.129:53505)
0620 09:30:06.729029 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0620 09:30:06.729043 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 09:30:06.729110 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:30:06.729318 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:30:06.729319 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:30:06.729321 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:30:06.729401 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:30:06.729403 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:30:06.730358 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:30:06.730359 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:30:06.731760 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:30:06.731764 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:30:06.732680 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:30:06.732681 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:30:06.732803 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:06.732805 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 09:30:06.732858 (+ 53us) client_negotiation.cc:240] Negotiation successful
0620 09:30:06.732913 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63,"mutex_wait_us":102}
I20260620 09:30:06.733000 3736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:06.728942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.130:33289 (local address 127.16.43.129:39117)
0620 09:30:06.729238 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0620 09:30:06.729246 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 09:30:06.729315 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:30:06.729449 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:30:06.729450 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:30:06.729452 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:30:06.729516 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:30:06.729517 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:30:06.730372 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:30:06.730372 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:30:06.731760 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:30:06.731764 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:30:06.732680 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:30:06.732681 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:30:06.732764 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:06.732770 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 09:30:06.732835 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 09:30:06.732898 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"mutex_wait_us":142,"thread_start_us":125,"threads_started":1}
I20260620 09:30:06.733459 3664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5147ef4a05de4a39a3db1fd89e775a54" candidate_uuid: "4b96223312784ef3820d2e8b2278e232" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60073351d4c243a88b215a4e5fbde0f8" is_pre_election: true
I20260620 09:30:06.733640 3664 raft_consensus.cc:2454] T 5147ef4a05de4a39a3db1fd89e775a54 P 60073351d4c243a88b215a4e5fbde0f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b96223312784ef3820d2e8b2278e232 in term 0.
I20260620 09:30:06.733666 3530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5147ef4a05de4a39a3db1fd89e775a54" candidate_uuid: "4b96223312784ef3820d2e8b2278e232" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d737dcf489d4fbfa03b962aaeeb557d" is_pre_election: true
I20260620 09:30:06.733876 3530 raft_consensus.cc:2454] T 5147ef4a05de4a39a3db1fd89e775a54 P 9d737dcf489d4fbfa03b962aaeeb557d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b96223312784ef3820d2e8b2278e232 in term 0.
I20260620 09:30:06.733901 3329 leader_election.cc:302] T 5147ef4a05de4a39a3db1fd89e775a54 P 4b96223312784ef3820d2e8b2278e232 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b96223312784ef3820d2e8b2278e232, 60073351d4c243a88b215a4e5fbde0f8; no voters:
I20260620 09:30:06.734046 3735 raft_consensus.cc:2781] T 5147ef4a05de4a39a3db1fd89e775a54 P 4b96223312784ef3820d2e8b2278e232 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 09:30:06.734107 3735 raft_consensus.cc:482] T 5147ef4a05de4a39a3db1fd89e775a54 P 4b96223312784ef3820d2e8b2278e232 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 09:30:06.734140 3735 raft_consensus.cc:3037] T 5147ef4a05de4a39a3db1fd89e775a54 P 4b96223312784ef3820d2e8b2278e232 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:30:06.734874 3735 raft_consensus.cc:504] T 5147ef4a05de4a39a3db1fd89e775a54 P 4b96223312784ef3820d2e8b2278e232 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d737dcf489d4fbfa03b962aaeeb557d" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 33289 } } peers { permanent_uuid: "60073351d4c243a88b215a4e5fbde0f8" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 34495 } } peers { permanent_uuid: "4b96223312784ef3820d2e8b2278e232" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 40815 } }
I20260620 09:30:06.735023 3735 leader_election.cc:288] T 5147ef4a05de4a39a3db1fd89e775a54 P 4b96223312784ef3820d2e8b2278e232 [CANDIDATE]: Term 1 election: Requested vote from peers 9d737dcf489d4fbfa03b962aaeeb557d (127.16.43.130:33289), 60073351d4c243a88b215a4e5fbde0f8 (127.16.43.131:34495)
I20260620 09:30:06.735186 3664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5147ef4a05de4a39a3db1fd89e775a54" candidate_uuid: "4b96223312784ef3820d2e8b2278e232" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60073351d4c243a88b215a4e5fbde0f8"
I20260620 09:30:06.735186 3530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5147ef4a05de4a39a3db1fd89e775a54" candidate_uuid: "4b96223312784ef3820d2e8b2278e232" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d737dcf489d4fbfa03b962aaeeb557d"
I20260620 09:30:06.735275 3530 raft_consensus.cc:3037] T 5147ef4a05de4a39a3db1fd89e775a54 P 9d737dcf489d4fbfa03b962aaeeb557d [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:30:06.735275 3664 raft_consensus.cc:3037] T 5147ef4a05de4a39a3db1fd89e775a54 P 60073351d4c243a88b215a4e5fbde0f8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:30:06.736210 3664 raft_consensus.cc:2454] T 5147ef4a05de4a39a3db1fd89e775a54 P 60073351d4c243a88b215a4e5fbde0f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b96223312784ef3820d2e8b2278e232 in term 1.
I20260620 09:30:06.736202 3530 raft_consensus.cc:2454] T 5147ef4a05de4a39a3db1fd89e775a54 P 9d737dcf489d4fbfa03b962aaeeb557d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b96223312784ef3820d2e8b2278e232 in term 1.
I20260620 09:30:06.736424 3329 leader_election.cc:302] T 5147ef4a05de4a39a3db1fd89e775a54 P 4b96223312784ef3820d2e8b2278e232 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b96223312784ef3820d2e8b2278e232, 60073351d4c243a88b215a4e5fbde0f8; no voters:
I20260620 09:30:06.736531 3735 raft_consensus.cc:2781] T 5147ef4a05de4a39a3db1fd89e775a54 P 4b96223312784ef3820d2e8b2278e232 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:30:06.736701 3735 raft_consensus.cc:686] T 5147ef4a05de4a39a3db1fd89e775a54 P 4b96223312784ef3820d2e8b2278e232 [term 1 LEADER]: Becoming Leader. State: Replica: 4b96223312784ef3820d2e8b2278e232, State: Running, Role: LEADER
I20260620 09:30:06.736800 3735 consensus_queue.cc:237] T 5147ef4a05de4a39a3db1fd89e775a54 P 4b96223312784ef3820d2e8b2278e232 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d737dcf489d4fbfa03b962aaeeb557d" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 33289 } } peers { permanent_uuid: "60073351d4c243a88b215a4e5fbde0f8" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 34495 } } peers { permanent_uuid: "4b96223312784ef3820d2e8b2278e232" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 40815 } }
I20260620 09:30:06.737735 3250 catalog_manager.cc:5495] T 5147ef4a05de4a39a3db1fd89e775a54 P 4b96223312784ef3820d2e8b2278e232 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4b96223312784ef3820d2e8b2278e232 (127.16.43.129). New cstate: current_term: 1 leader_uuid: "4b96223312784ef3820d2e8b2278e232" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d737dcf489d4fbfa03b962aaeeb557d" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 33289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "60073351d4c243a88b215a4e5fbde0f8" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 34495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4b96223312784ef3820d2e8b2278e232" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 40815 } health_report { overall_health: HEALTHY } } }
I20260620 09:30:06.749935 3724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:06.747335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40678 (local address 127.16.43.129:40815)
0620 09:30:06.747439 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:06.747440 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:06.747447 (+ 7us) server_negotiation.cc:407] Connection header received
0620 09:30:06.747481 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:06.747481 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:06.747501 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:06.747618 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:30:06.747769 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:06.748335 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:06.749096 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:06.749325 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:06.749403 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:06.749576 (+ 173us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:30:06.749690 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:30:06.749790 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:06.749809 (+ 19us) server_negotiation.cc:299] Negotiation successful
0620 09:30:06.749830 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 09:30:06.751711 3530 raft_consensus.cc:1261] T 5147ef4a05de4a39a3db1fd89e775a54 P 9d737dcf489d4fbfa03b962aaeeb557d [term 1 FOLLOWER]: Refusing update from remote peer 4b96223312784ef3820d2e8b2278e232: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:30:06.751714 3664 raft_consensus.cc:1261] T 5147ef4a05de4a39a3db1fd89e775a54 P 60073351d4c243a88b215a4e5fbde0f8 [term 1 FOLLOWER]: Refusing update from remote peer 4b96223312784ef3820d2e8b2278e232: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:30:06.752038 3737 consensus_queue.cc:1035] T 5147ef4a05de4a39a3db1fd89e775a54 P 4b96223312784ef3820d2e8b2278e232 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d737dcf489d4fbfa03b962aaeeb557d" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 33289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:30:06.752184 3735 consensus_queue.cc:1035] T 5147ef4a05de4a39a3db1fd89e775a54 P 4b96223312784ef3820d2e8b2278e232 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60073351d4c243a88b215a4e5fbde0f8" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 34495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:30:06.755020 3742 mvcc.cc:205] Tried to move back new op lower bound from 7298858216452161536 to 7298858216394461184. Current Snapshot: MvccSnapshot[applied={T|T < 7298858216452161536}]
I20260620 09:30:06.755100 3740 mvcc.cc:205] Tried to move back new op lower bound from 7298858216452161536 to 7298858216394461184. Current Snapshot: MvccSnapshot[applied={T|T < 7298858216452161536}]
I20260620 09:30:06.755275 3741 mvcc.cc:205] Tried to move back new op lower bound from 7298858216452161536 to 7298858216394461184. Current Snapshot: MvccSnapshot[applied={T|T < 7298858216452161536}]
I20260620 09:30:06.757604 3250 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48950:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:30:06.757716 3250 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48950:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:30:06.758450 3250 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c119f98246eb4308a054f02b32a59163: Sending DeleteTablet for 3 replicas of tablet 5147ef4a05de4a39a3db1fd89e775a54
I20260620 09:30:06.758832 3644 tablet_service.cc:1514] Processing DeleteTablet for tablet 5147ef4a05de4a39a3db1fd89e775a54 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:30:06 UTC) from {username='kudu', principal='kudu/127.16.43.190@KRBTEST.COM'} at 127.0.0.1:43254
I20260620 09:30:06.758819 3510 tablet_service.cc:1514] Processing DeleteTablet for tablet 5147ef4a05de4a39a3db1fd89e775a54 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:30:06 UTC) from {username='kudu', principal='kudu/127.16.43.190@KRBTEST.COM'} at 127.0.0.1:37946
I20260620 09:30:06.759048 3376 tablet_service.cc:1514] Processing DeleteTablet for tablet 5147ef4a05de4a39a3db1fd89e775a54 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:30:06 UTC) from {username='kudu', principal='kudu/127.16.43.190@KRBTEST.COM'} at 127.0.0.1:40662
I20260620 09:30:06.759289 3752 tablet_replica.cc:330] T 5147ef4a05de4a39a3db1fd89e775a54 P 4b96223312784ef3820d2e8b2278e232: stopping tablet replica
I20260620 09:30:06.759431 3752 raft_consensus.cc:2229] T 5147ef4a05de4a39a3db1fd89e775a54 P 4b96223312784ef3820d2e8b2278e232 [term 1 LEADER]: Raft consensus shutting down.
I20260620 09:30:06.759471 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 3313
I20260620 09:30:06.759315 3751 tablet_replica.cc:330] T 5147ef4a05de4a39a3db1fd89e775a54 P 9d737dcf489d4fbfa03b962aaeeb557d: stopping tablet replica
I20260620 09:30:06.764319 3751 raft_consensus.cc:2229] T 5147ef4a05de4a39a3db1fd89e775a54 P 9d737dcf489d4fbfa03b962aaeeb557d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:30:06.764400 3750 tablet_replica.cc:330] T 5147ef4a05de4a39a3db1fd89e775a54 P 60073351d4c243a88b215a4e5fbde0f8: stopping tablet replica
I20260620 09:30:06.764464 3751 raft_consensus.cc:2258] T 5147ef4a05de4a39a3db1fd89e775a54 P 9d737dcf489d4fbfa03b962aaeeb557d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:30:06.764513 3750 raft_consensus.cc:2229] T 5147ef4a05de4a39a3db1fd89e775a54 P 60073351d4c243a88b215a4e5fbde0f8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:30:06.764608 3750 raft_consensus.cc:2258] T 5147ef4a05de4a39a3db1fd89e775a54 P 60073351d4c243a88b215a4e5fbde0f8 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260620 09:30:06.765027 3237 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.43.129:40815 (error 108)
I20260620 09:30:06.765154 3750 ts_tablet_manager.cc:1905] T 5147ef4a05de4a39a3db1fd89e775a54 P 60073351d4c243a88b215a4e5fbde0f8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:30:06.765259 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 3447
I20260620 09:30:06.766861 3721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:06.765930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.129:40815 (local address 127.0.0.1:40688)
0620 09:30:06.766757 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0620 09:30:06.766811 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.129:40815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":797}
I20260620 09:30:06.767249 3750 ts_tablet_manager.cc:1918] T 5147ef4a05de4a39a3db1fd89e775a54 P 60073351d4c243a88b215a4e5fbde0f8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:30:06.767343 3750 log.cc:1196] T 5147ef4a05de4a39a3db1fd89e775a54 P 60073351d4c243a88b215a4e5fbde0f8: Deleting WAL directory at /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781947562949448-16558-0/minicluster-data/ts-2/wal/wals/5147ef4a05de4a39a3db1fd89e775a54
I20260620 09:30:06.767642 3750 ts_tablet_manager.cc:1939] T 5147ef4a05de4a39a3db1fd89e775a54 P 60073351d4c243a88b215a4e5fbde0f8: Deleting consensus metadata
I20260620 09:30:06.768091 3235 catalog_manager.cc:4841] TS 60073351d4c243a88b215a4e5fbde0f8 (127.16.43.131:34495): tablet 5147ef4a05de4a39a3db1fd89e775a54 (table test-table [id=2277900927e24585970d79ec0424297f]) successfully deleted
W20260620 09:30:06.768147 3237 catalog_manager.cc:4584] TS 4b96223312784ef3820d2e8b2278e232 (127.16.43.129:40815): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5147ef4a05de4a39a3db1fd89e775a54: Network error: Client connection negotiation failed: client connection to 127.16.43.129:40815: connect: Connection refused (error 111)
I20260620 09:30:06.770522 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 3581
I20260620 09:30:06.770597 3721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:06.770457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.130:33289 (local address 127.0.0.1:37960)
0620 09:30:06.770526 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0620 09:30:06.770559 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.130:33289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260620 09:30:06.775121 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 3219
2026-06-20T09:30:06Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2021 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-20T09:30:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:30:06Z Disabled control of system clock
I20260620 09:30:06.804244 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781947562949448-16558-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781947562949448-16558-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781947562949448-16558-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.43.190:45093
--webserver_interface=127.16.43.190
--webserver_port=0
--builtin_ntp_servers=127.16.43.148:45285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.43.190:45093
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 09:30:06.892637 3757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:30:06.892938 3757 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:30:06.892976 3757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:30:06.894169 3757 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:30:06.894244 3757 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:30:06.894265 3757 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:30:06.894285 3757 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:30:06.894312 3757 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:30:06.895813 3757 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:45285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781947562949448-16558-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781947562949448-16558-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.43.190:45093
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.43.190:45093
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.43.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3757
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781947562949448-16558-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:30:06.896099 3757 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:30:06.896435 3757 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:30:06.902215 3763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:30:06.902248 3762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:30:06.902289 3765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:30:06.902288 3757 server_base.cc:1034] running on GCE node
I20260620 09:30:06.902702 3757 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:30:06.903069 3757 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:30:06.904227 3757 hybrid_clock.cc:648] HybridClock initialized: now 1781947806904203 us; error 40 us; skew 500 ppm
I20260620 09:30:06.905927 3757 webserver.cc:460] Webserver started at http://127.16.43.190:36369/ using document root <none> and password file <none>
I20260620 09:30:06.906199 3757 fs_manager.cc:359] Metadata directory not provided
I20260620 09:30:06.906272 3757 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:30:06.906425 3757 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:30:06.907668 3757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781947562949448-16558-0/minicluster-data/master-0/data/instance:
uuid: "5716b3c1340c4f2bba7c09ae276e52b4"
format_stamp: "Formatted at 2026-06-20 09:30:06 on dist-test-slave-706q"
server_key: "7c2606672a3a6489f0ff700bf62488db"
server_key_iv: "3843371baae1952849052f54bcaf08f9"
server_key_version: "encryptionkey@0"
I20260620 09:30:06.908062 3757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781947562949448-16558-0/minicluster-data/master-0/wal/instance:
uuid: "5716b3c1340c4f2bba7c09ae276e52b4"
format_stamp: "Formatted at 2026-06-20 09:30:06 on dist-test-slave-706q"
server_key: "7c2606672a3a6489f0ff700bf62488db"
server_key_iv: "3843371baae1952849052f54bcaf08f9"
server_key_version: "encryptionkey@0"
I20260620 09:30:06.909592 3757 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 09:30:06.910468 3771 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:30:06.910815 3757 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:30:06.910929 3757 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781947562949448-16558-0/minicluster-data/master-0/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781947562949448-16558-0/minicluster-data/master-0/wal
uuid: "5716b3c1340c4f2bba7c09ae276e52b4"
format_stamp: "Formatted at 2026-06-20 09:30:06 on dist-test-slave-706q"
server_key: "7c2606672a3a6489f0ff700bf62488db"
server_key_iv: "3843371baae1952849052f54bcaf08f9"
server_key_version: "encryptionkey@0"
I20260620 09:30:06.911036 3757 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781947562949448-16558-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781947562949448-16558-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781947562949448-16558-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:30:06.948689 3757 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:30:06.949071 3757 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:30:06.949226 3757 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:30:06.953559 3757 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.190:45093
I20260620 09:30:06.953600 3823 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.190:45093 every 8 connection(s)
I20260620 09:30:06.953955 3757 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
I20260620 09:30:06.955016 3824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:30:06.957406 3824 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5716b3c1340c4f2bba7c09ae276e52b4: Bootstrap starting.
I20260620 09:30:06.958045 3824 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5716b3c1340c4f2bba7c09ae276e52b4: Neither blocks nor log segments found. Creating new log.
I20260620 09:30:06.958359 3824 log.cc:824] T 00000000000000000000000000000000 P 5716b3c1340c4f2bba7c09ae276e52b4: Log is configured to *not* fsync() on all Append() calls
I20260620 09:30:06.959126 3824 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5716b3c1340c4f2bba7c09ae276e52b4: No bootstrap required, opened a new log
I20260620 09:30:06.960278 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 3757
I20260620 09:30:06.960397 16558 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781947562949448-16558-0/minicluster-data/master-0/wal/instance
I20260620 09:30:06.960605 16558 external_mini_cluster.cc:1428] Setting key 560c2c4d00104ea3dad55a21dc0ea2f1
I20260620 09:30:06.961431 3824 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5716b3c1340c4f2bba7c09ae276e52b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5716b3c1340c4f2bba7c09ae276e52b4" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 45093 } }
I20260620 09:30:06.961596 3824 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5716b3c1340c4f2bba7c09ae276e52b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:30:06.961651 3824 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5716b3c1340c4f2bba7c09ae276e52b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5716b3c1340c4f2bba7c09ae276e52b4, State: Initialized, Role: FOLLOWER
I20260620 09:30:06.961820 3824 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5716b3c1340c4f2bba7c09ae276e52b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5716b3c1340c4f2bba7c09ae276e52b4" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 45093 } }
I20260620 09:30:06.961910 3824 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5716b3c1340c4f2bba7c09ae276e52b4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:30:06.961952 3824 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5716b3c1340c4f2bba7c09ae276e52b4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:30:06.962010 3824 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5716b3c1340c4f2bba7c09ae276e52b4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:30:06.962822 3824 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5716b3c1340c4f2bba7c09ae276e52b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5716b3c1340c4f2bba7c09ae276e52b4" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 45093 } }
I20260620 09:30:06.962935 3824 leader_election.cc:302] T 00000000000000000000000000000000 P 5716b3c1340c4f2bba7c09ae276e52b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5716b3c1340c4f2bba7c09ae276e52b4; no voters:
I20260620 09:30:06.963094 3824 leader_election.cc:288] T 00000000000000000000000000000000 P 5716b3c1340c4f2bba7c09ae276e52b4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:30:06.963210 3829 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5716b3c1340c4f2bba7c09ae276e52b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:30:06.963361 3829 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5716b3c1340c4f2bba7c09ae276e52b4 [term 1 LEADER]: Becoming Leader. State: Replica: 5716b3c1340c4f2bba7c09ae276e52b4, State: Running, Role: LEADER
I20260620 09:30:06.963457 3824 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5716b3c1340c4f2bba7c09ae276e52b4 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:30:06.963449 3829 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5716b3c1340c4f2bba7c09ae276e52b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5716b3c1340c4f2bba7c09ae276e52b4" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 45093 } }
I20260620 09:30:06.963871 3830 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5716b3c1340c4f2bba7c09ae276e52b4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5716b3c1340c4f2bba7c09ae276e52b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5716b3c1340c4f2bba7c09ae276e52b4" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 45093 } } }
I20260620 09:30:06.963960 3830 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5716b3c1340c4f2bba7c09ae276e52b4 [sys.catalog]: This master's current role is: LEADER
I20260620 09:30:06.964218 3831 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5716b3c1340c4f2bba7c09ae276e52b4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5716b3c1340c4f2bba7c09ae276e52b4. Latest consensus state: current_term: 1 leader_uuid: "5716b3c1340c4f2bba7c09ae276e52b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5716b3c1340c4f2bba7c09ae276e52b4" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 45093 } } }
I20260620 09:30:06.964336 3831 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5716b3c1340c4f2bba7c09ae276e52b4 [sys.catalog]: This master's current role is: LEADER
I20260620 09:30:06.964540 3835 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:30:06.965067 3835 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:30:06.966694 3835 catalog_manager.cc:1347] Generated new cluster ID: cda8aac418314501855f5702fbf9f129
I20260620 09:30:06.966751 3835 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:30:06.967244 3828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:06.961188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39788 (local address 127.16.43.190:45093)
0620 09:30:06.961421 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:06.961434 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:06.961459 (+ 25us) server_negotiation.cc:407] Connection header received
0620 09:30:06.961559 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:06.961565 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:06.961618 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:06.961794 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:30:06.962048 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:06.963851 (+ 1803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:06.965005 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:06.965657 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:06.965804 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:06.966497 (+ 693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:30:06.966501 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:30:06.966502 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:30:06.966722 (+ 220us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:30:06.966784 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:30:06.966800 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:30:06.966863 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:06.966945 (+ 82us) server_negotiation.cc:299] Negotiation successful
0620 09:30:06.967029 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":111,"threads_started":1}
I20260620 09:30:06.977077 3835 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:30:06.977628 3835 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:30:06.982210 3835 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5716b3c1340c4f2bba7c09ae276e52b4: Generated new TSK 0
I20260620 09:30:06.982420 3835 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 09:30:06.988135 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781947562949448-16558-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781947562949448-16558-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781947562949448-16558-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781947562949448-16558-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.43.129:0
--local_ip_for_outbound_sockets=127.16.43.129
--webserver_interface=127.16.43.129
--webserver_port=0
--tserver_master_addrs=127.16.43.190:45093
--builtin_ntp_servers=127.16.43.148:45285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 09:30:07.077226 3848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:30:07.077474 3848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:30:07.077512 3848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:30:07.078706 3848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:30:07.078768 3848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:30:07.078811 3848 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.43.129
I20260620 09:30:07.080237 3848 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:45285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781947562949448-16558-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781947562949448-16558-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.43.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781947562949448-16558-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.43.129
--webserver_port=0
--tserver_master_addrs=127.16.43.190:45093
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3848
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.43.129
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781947562949448-16558-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:30:07.080551 3848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:30:07.080849 3848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:30:07.086830 3856 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:30:07.086804 3854 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:30:07.087076 3848 server_base.cc:1034] running on GCE node
W20260620 09:30:07.086856 3853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:30:07.087332 3848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:30:07.087627 3848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:30:07.088783 3848 hybrid_clock.cc:648] HybridClock initialized: now 1781947807088772 us; error 37 us; skew 500 ppm
I20260620 09:30:07.090070 3848 webserver.cc:460] Webserver started at http://127.16.43.129:33879/ using document root <none> and password file <none>
I20260620 09:30:07.090339 3848 fs_manager.cc:359] Metadata directory not provided
I20260620 09:30:07.090409 3848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:30:07.090560 3848 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:30:07.091699 3848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781947562949448-16558-0/minicluster-data/ts-0/data/instance:
uuid: "c168e12565444f959606259299318f32"
format_stamp: "Formatted at 2026-06-20 09:30:07 on dist-test-slave-706q"
server_key: "df07845bebcbdf0ef1eb2cd1f7af65ed"
server_key_iv: "9d54a5a913eee4119d1cdbf83a7d16d4"
server_key_version: "encryptionkey@0"
I20260620 09:30:07.092092 3848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781947562949448-16558-0/minicluster-data/ts-0/wal/instance:
uuid: "c168e12565444f959606259299318f32"
format_stamp: "Formatted at 2026-06-20 09:30:07 on dist-test-slave-706q"
server_key: "df07845bebcbdf0ef1eb2cd1f7af65ed"
server_key_iv: "9d54a5a913eee4119d1cdbf83a7d16d4"
server_key_version: "encryptionkey@0"
I20260620 09:30:07.093432 3848 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 09:30:07.094190 3862 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:30:07.094354 3848 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 09:30:07.094435 3848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781947562949448-16558-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781947562949448-16558-0/minicluster-data/ts-0/wal
uuid: "c168e12565444f959606259299318f32"
format_stamp: "Formatted at 2026-06-20 09:30:07 on dist-test-slave-706q"
server_key: "df07845bebcbdf0ef1eb2cd1f7af65ed"
server_key_iv: "9d54a5a913eee4119d1cdbf83a7d16d4"
server_key_version: "encryptionkey@0"
I20260620 09:30:07.094523 3848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781947562949448-16558-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781947562949448-16558-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781947562949448-16558-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:30:07.108335 3848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:30:07.108652 3848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:30:07.108804 3848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:30:07.109033 3848 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:30:07.109375 3848 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:30:07.109426 3848 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:30:07.109468 3848 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:30:07.109498 3848 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:30:07.115397 3848 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.129:35391
I20260620 09:30:07.115495 3975 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.129:35391 every 8 connection(s)
I20260620 09:30:07.115911 3848 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781947562949448-16558-0/minicluster-data/ts-0/data/info.pb
I20260620 09:30:07.121537 3828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:07.116664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.129:57465 (local address 127.16.43.190:45093)
0620 09:30:07.116748 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:07.116750 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:07.116956 (+ 206us) server_negotiation.cc:407] Connection header received
0620 09:30:07.117489 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:07.117491 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:07.117504 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:07.117560 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:30:07.118323 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:07.118953 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:07.120096 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:07.120291 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:07.120353 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:07.121243 (+ 890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:30:07.121244 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:30:07.121245 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:30:07.121250 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:30:07.121264 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:30:07.121268 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:30:07.121344 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:07.121442 (+ 98us) server_negotiation.cc:299] Negotiation successful
0620 09:30:07.121473 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 09:30:07.121757 3978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:07.116622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:45093 (local address 127.16.43.129:57465)
0620 09:30:07.116824 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0620 09:30:07.116861 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 09:30:07.117396 (+ 535us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:30:07.117618 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:30:07.117621 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:30:07.117630 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:30:07.118206 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:30:07.118212 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:30:07.119067 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:30:07.119068 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:30:07.120004 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:30:07.120006 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:30:07.120374 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:30:07.120375 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:30:07.120491 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:07.121038 (+ 547us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 09:30:07.121038 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:30:07.121068 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 09:30:07.121091 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:30:07.121367 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:30:07.121372 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:30:07.121375 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 09:30:07.121437 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 09:30:07.121526 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":99,"threads_started":1}
I20260620 09:30:07.122309 3976 heartbeater.cc:347] Connected to a master server at 127.16.43.190:45093
I20260620 09:30:07.122396 3976 heartbeater.cc:464] Registering TS with master...
I20260620 09:30:07.122604 3976 heartbeater.cc:511] Master 127.16.43.190:45093 requested a full tablet report, sending...
I20260620 09:30:07.123162 3788 ts_manager.cc:194] Registered new tserver with Master: c168e12565444f959606259299318f32 (127.16.43.129:35391)
I20260620 09:30:07.123907 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 3848
I20260620 09:30:07.123901 3788 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.43.129:57465
I20260620 09:30:07.124017 16558 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781947562949448-16558-0/minicluster-data/ts-0/wal/instance
I20260620 09:30:07.124233 16558 external_mini_cluster.cc:1428] Setting key f52dae71c1e1f524dbc106fbdd854fc7
I20260620 09:30:07.125129 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781947562949448-16558-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781947562949448-16558-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781947562949448-16558-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781947562949448-16558-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.43.130:0
--local_ip_for_outbound_sockets=127.16.43.130
--webserver_interface=127.16.43.130
--webserver_port=0
--tserver_master_addrs=127.16.43.190:45093
--builtin_ntp_servers=127.16.43.148:45285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 09:30:07.208086 3979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:30:07.208323 3979 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:30:07.208348 3979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:30:07.209581 3979 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:30:07.209630 3979 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:30:07.209659 3979 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.43.130
I20260620 09:30:07.211167 3979 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:45285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781947562949448-16558-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781947562949448-16558-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.43.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781947562949448-16558-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.43.130
--webserver_port=0
--tserver_master_addrs=127.16.43.190:45093
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3979
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.43.130
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781947562949448-16558-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:30:07.211426 3979 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:30:07.211758 3979 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:30:07.216915 3987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:30:07.216986 3985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:30:07.217111 3979 server_base.cc:1034] running on GCE node
W20260620 09:30:07.216982 3984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:30:07.217442 3979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:30:07.217651 3979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:30:07.218794 3979 hybrid_clock.cc:648] HybridClock initialized: now 1781947807218771 us; error 33 us; skew 500 ppm
I20260620 09:30:07.220213 3979 webserver.cc:460] Webserver started at http://127.16.43.130:40191/ using document root <none> and password file <none>
I20260620 09:30:07.220477 3979 fs_manager.cc:359] Metadata directory not provided
I20260620 09:30:07.220523 3979 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:30:07.220680 3979 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:30:07.221738 3979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781947562949448-16558-0/minicluster-data/ts-1/data/instance:
uuid: "9223be19b5b64d089215a99f669e2f40"
format_stamp: "Formatted at 2026-06-20 09:30:07 on dist-test-slave-706q"
server_key: "eb4e781b90fd343f2948674c0d937234"
server_key_iv: "aee1446380803a1bade1b9e6d3d4b7e0"
server_key_version: "encryptionkey@0"
I20260620 09:30:07.222082 3979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781947562949448-16558-0/minicluster-data/ts-1/wal/instance:
uuid: "9223be19b5b64d089215a99f669e2f40"
format_stamp: "Formatted at 2026-06-20 09:30:07 on dist-test-slave-706q"
server_key: "eb4e781b90fd343f2948674c0d937234"
server_key_iv: "aee1446380803a1bade1b9e6d3d4b7e0"
server_key_version: "encryptionkey@0"
I20260620 09:30:07.223340 3979 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 09:30:07.224155 3993 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:30:07.224354 3979 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 09:30:07.224438 3979 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781947562949448-16558-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781947562949448-16558-0/minicluster-data/ts-1/wal
uuid: "9223be19b5b64d089215a99f669e2f40"
format_stamp: "Formatted at 2026-06-20 09:30:07 on dist-test-slave-706q"
server_key: "eb4e781b90fd343f2948674c0d937234"
server_key_iv: "aee1446380803a1bade1b9e6d3d4b7e0"
server_key_version: "encryptionkey@0"
I20260620 09:30:07.224510 3979 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781947562949448-16558-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781947562949448-16558-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781947562949448-16558-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:30:07.233664 3979 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:30:07.233979 3979 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:30:07.234129 3979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:30:07.234359 3979 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:30:07.234679 3979 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:30:07.234714 3979 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:30:07.234736 3979 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:30:07.234787 3979 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:30:07.240866 3979 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.130:43723
I20260620 09:30:07.240912 4106 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.130:43723 every 8 connection(s)
I20260620 09:30:07.241281 3979 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781947562949448-16558-0/minicluster-data/ts-1/data/info.pb
I20260620 09:30:07.249125 3828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:07.242086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.130:41535 (local address 127.16.43.190:45093)
0620 09:30:07.242132 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:07.242134 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:07.242367 (+ 233us) server_negotiation.cc:407] Connection header received
0620 09:30:07.242952 (+ 585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:07.242954 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:07.242971 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:07.243060 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:30:07.244166 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:07.245081 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:07.247402 (+ 2321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:07.247673 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:07.247736 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:07.248768 (+ 1032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:30:07.248769 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:30:07.248770 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:30:07.248774 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:30:07.248792 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:30:07.248796 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:30:07.248850 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:07.249021 (+ 171us) server_negotiation.cc:299] Negotiation successful
0620 09:30:07.249051 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 09:30:07.249486 4109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:07.242048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:45093 (local address 127.16.43.130:41535)
0620 09:30:07.242242 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0620 09:30:07.242284 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 09:30:07.242805 (+ 521us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:30:07.243215 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:30:07.243220 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:30:07.243233 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:30:07.244019 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:30:07.244026 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:30:07.246274 (+ 2248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:30:07.246275 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:30:07.247315 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:30:07.247318 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:30:07.247760 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:30:07.247762 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:30:07.247945 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:07.248571 (+ 626us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 09:30:07.248572 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:30:07.248616 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 09:30:07.248650 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:30:07.248917 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:30:07.248925 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:30:07.248929 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 09:30:07.249020 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 09:30:07.249185 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":118,"threads_started":1}
I20260620 09:30:07.250226 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 3979
I20260620 09:30:07.250195 4107 heartbeater.cc:347] Connected to a master server at 127.16.43.190:45093
I20260620 09:30:07.250336 4107 heartbeater.cc:464] Registering TS with master...
I20260620 09:30:07.250342 16558 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781947562949448-16558-0/minicluster-data/ts-1/wal/instance
I20260620 09:30:07.250548 16558 external_mini_cluster.cc:1428] Setting key c1645231bad71e1503624d6627b9581e
I20260620 09:30:07.250571 4107 heartbeater.cc:511] Master 127.16.43.190:45093 requested a full tablet report, sending...
I20260620 09:30:07.250998 3787 ts_manager.cc:194] Registered new tserver with Master: 9223be19b5b64d089215a99f669e2f40 (127.16.43.130:43723)
I20260620 09:30:07.251474 3787 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.43.130:41535
I20260620 09:30:07.251710 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781947562949448-16558-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781947562949448-16558-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781947562949448-16558-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781947562949448-16558-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.43.131:0
--local_ip_for_outbound_sockets=127.16.43.131
--webserver_interface=127.16.43.131
--webserver_port=0
--tserver_master_addrs=127.16.43.190:45093
--builtin_ntp_servers=127.16.43.148:45285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 09:30:07.335758 4110 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:30:07.336017 4110 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:30:07.336045 4110 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:30:07.337218 4110 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:30:07.337265 4110 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:30:07.337292 4110 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.43.131
I20260620 09:30:07.338972 4110 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:45285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781947562949448-16558-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781947562949448-16558-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.43.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781947562949448-16558-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.43.131
--webserver_port=0
--tserver_master_addrs=127.16.43.190:45093
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4110
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.43.131
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781947562949448-16558-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:30:07.339296 4110 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:30:07.339695 4110 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:30:07.344981 4116 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:30:07.345021 4115 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:30:07.345057 4110 server_base.cc:1034] running on GCE node
W20260620 09:30:07.345022 4118 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:30:07.345378 4110 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:30:07.345709 4110 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:30:07.346877 4110 hybrid_clock.cc:648] HybridClock initialized: now 1781947807346863 us; error 42 us; skew 500 ppm
I20260620 09:30:07.348228 4110 webserver.cc:460] Webserver started at http://127.16.43.131:44511/ using document root <none> and password file <none>
I20260620 09:30:07.348474 4110 fs_manager.cc:359] Metadata directory not provided
I20260620 09:30:07.348544 4110 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:30:07.348735 4110 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:30:07.349834 4110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781947562949448-16558-0/minicluster-data/ts-2/data/instance:
uuid: "aa49757dd7d54a9fbe9c3f1ddcba70b0"
format_stamp: "Formatted at 2026-06-20 09:30:07 on dist-test-slave-706q"
server_key: "e4cc51232eee962dea10a39f2ebed6f3"
server_key_iv: "6bfd967298add69644985935a657468b"
server_key_version: "encryptionkey@0"
I20260620 09:30:07.350199 4110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781947562949448-16558-0/minicluster-data/ts-2/wal/instance:
uuid: "aa49757dd7d54a9fbe9c3f1ddcba70b0"
format_stamp: "Formatted at 2026-06-20 09:30:07 on dist-test-slave-706q"
server_key: "e4cc51232eee962dea10a39f2ebed6f3"
server_key_iv: "6bfd967298add69644985935a657468b"
server_key_version: "encryptionkey@0"
I20260620 09:30:07.351816 4110 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 09:30:07.352576 4124 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:30:07.352756 4110 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 09:30:07.352855 4110 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781947562949448-16558-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781947562949448-16558-0/minicluster-data/ts-2/wal
uuid: "aa49757dd7d54a9fbe9c3f1ddcba70b0"
format_stamp: "Formatted at 2026-06-20 09:30:07 on dist-test-slave-706q"
server_key: "e4cc51232eee962dea10a39f2ebed6f3"
server_key_iv: "6bfd967298add69644985935a657468b"
server_key_version: "encryptionkey@0"
I20260620 09:30:07.352936 4110 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781947562949448-16558-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781947562949448-16558-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781947562949448-16558-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:30:07.384728 4110 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:30:07.385080 4110 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:30:07.385233 4110 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:30:07.385483 4110 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:30:07.385847 4110 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:30:07.385901 4110 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:30:07.385944 4110 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:30:07.385977 4110 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:30:07.391942 4110 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.131:44289
I20260620 09:30:07.391984 4237 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.131:44289 every 8 connection(s)
I20260620 09:30:07.392314 4110 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781947562949448-16558-0/minicluster-data/ts-2/data/info.pb
I20260620 09:30:07.396847 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 4110
I20260620 09:30:07.396967 16558 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781947562949448-16558-0/minicluster-data/ts-2/wal/instance
I20260620 09:30:07.397204 16558 external_mini_cluster.cc:1428] Setting key cee67b0904c4bc07c03a89b50494fcd9
I20260620 09:30:07.397596 3828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:07.393012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.131:54177 (local address 127.16.43.190:45093)
0620 09:30:07.393112 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:07.393115 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:07.393309 (+ 194us) server_negotiation.cc:407] Connection header received
0620 09:30:07.393776 (+ 467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:07.393777 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:07.393788 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:07.393842 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:30:07.394484 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:07.395011 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:07.396073 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:07.396267 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:07.396319 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:07.397216 (+ 897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:30:07.397218 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:30:07.397218 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:30:07.397223 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:30:07.397235 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:30:07.397239 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:30:07.397346 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:07.397448 (+ 102us) server_negotiation.cc:299] Negotiation successful
0620 09:30:07.397505 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 09:30:07.397821 4240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:07.392970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:45093 (local address 127.16.43.131:54177)
0620 09:30:07.393177 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0620 09:30:07.393212 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 09:30:07.393689 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:30:07.393903 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:30:07.393906 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:30:07.393914 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:30:07.394411 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:30:07.394416 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:30:07.395088 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:30:07.395089 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:30:07.395992 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:30:07.395994 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:30:07.396347 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:30:07.396347 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:30:07.396451 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:07.397027 (+ 576us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 09:30:07.397028 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:30:07.397069 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 09:30:07.397101 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:30:07.397348 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:30:07.397354 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:30:07.397358 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 09:30:07.397433 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 09:30:07.397550 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":104,"threads_started":1}
I20260620 09:30:07.398366 4238 heartbeater.cc:347] Connected to a master server at 127.16.43.190:45093
I20260620 09:30:07.398494 4238 heartbeater.cc:464] Registering TS with master...
I20260620 09:30:07.398696 4238 heartbeater.cc:511] Master 127.16.43.190:45093 requested a full tablet report, sending...
I20260620 09:30:07.399076 3787 ts_manager.cc:194] Registered new tserver with Master: aa49757dd7d54a9fbe9c3f1ddcba70b0 (127.16.43.131:44289)
I20260620 09:30:07.399423 3787 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.43.131:54177
I20260620 09:30:07.409637 16558 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 09:30:07.414408 3828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:07.411233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39792 (local address 127.16.43.190:45093)
0620 09:30:07.411316 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:07.411319 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:07.411412 (+ 93us) server_negotiation.cc:407] Connection header received
0620 09:30:07.411500 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:07.411501 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:07.411515 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:07.411608 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:30:07.411779 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:07.412276 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:07.413094 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:07.413305 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:07.413388 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:07.414127 (+ 739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:30:07.414129 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:30:07.414129 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:30:07.414135 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:30:07.414149 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:30:07.414153 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:30:07.414221 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:07.414278 (+ 57us) server_negotiation.cc:299] Negotiation successful
0620 09:30:07.414305 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 09:30:07.416323 3787 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39792:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 09:30:07.417930 3787 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 09:30:07.424721 4249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:07.420455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59432 (local address 127.16.43.131:44289)
0620 09:30:07.420749 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:07.420754 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:07.420766 (+ 12us) server_negotiation.cc:407] Connection header received
0620 09:30:07.420810 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:07.420813 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:07.420830 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:07.420927 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:30:07.421101 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:07.422287 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:07.423586 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:07.424241 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:07.424349 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:07.424465 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:07.424570 (+ 105us) server_negotiation.cc:299] Negotiation successful
0620 09:30:07.424614 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":79,"threads_started":1}
I20260620 09:30:07.424808 4248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:07.420287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.131:44289 (local address 127.0.0.1:59432)
0620 09:30:07.420494 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0620 09:30:07.420537 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 09:30:07.420607 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:30:07.420939 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:30:07.420942 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:30:07.420945 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:30:07.421020 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:30:07.421023 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:30:07.422368 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:30:07.422369 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:30:07.423463 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:30:07.423466 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:30:07.424354 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:30:07.424355 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:30:07.424455 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:07.424473 (+ 18us) client_negotiation.cc:769] Sending connection context
0620 09:30:07.424550 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 09:30:07.424693 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"mutex_wait_us":2,"thread_start_us":156,"threads_started":1}
I20260620 09:30:07.425057 4247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:07.420287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.129:35391 (local address 127.0.0.1:44106)
0620 09:30:07.420523 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0620 09:30:07.420537 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 09:30:07.420607 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:30:07.421025 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:30:07.421026 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:30:07.421028 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:30:07.421115 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:30:07.421116 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:30:07.422155 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:30:07.422156 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:30:07.423463 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:30:07.423466 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:30:07.424508 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:30:07.424509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:30:07.424570 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:07.424574 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 09:30:07.424619 (+ 45us) client_negotiation.cc:240] Negotiation successful
0620 09:30:07.424660 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"mutex_wait_us":48,"thread_start_us":65,"threads_started":1}
I20260620 09:30:07.425357 4251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:07.420626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44106 (local address 127.16.43.129:35391)
0620 09:30:07.420799 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:07.420804 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:07.420814 (+ 10us) server_negotiation.cc:407] Connection header received
0620 09:30:07.420852 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:07.420854 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:07.420868 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:07.420962 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:30:07.421171 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:07.422066 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:07.423598 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:07.424321 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:07.424392 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:07.424475 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:07.424741 (+ 266us) server_negotiation.cc:299] Negotiation successful
0620 09:30:07.425270 (+ 529us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":87,"threads_started":1}
I20260620 09:30:07.425655 4172 tablet_service.cc:1467] Processing CreateTablet for tablet 4fea0b97f8e947978bc57b749f2a4774 (DEFAULT_TABLE table=test-table [id=fc65cb4dfe844f6dab9e13407bc297d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:30:07.426023 4246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:07.420287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.130:43723 (local address 127.0.0.1:40688)
0620 09:30:07.420722 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0620 09:30:07.420732 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 09:30:07.420772 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:30:07.421019 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:30:07.421020 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:30:07.421022 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:30:07.421125 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:30:07.421128 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:30:07.423592 (+ 2464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:30:07.423592 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:30:07.424654 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:30:07.424656 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:30:07.425777 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:30:07.425778 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:30:07.425826 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:07.425830 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 09:30:07.425910 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 09:30:07.425953 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":423,"mutex_wait_us":3,"thread_start_us":62,"threads_started":1}
I20260620 09:30:07.426049 4250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:07.420538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40688 (local address 127.16.43.130:43723)
0620 09:30:07.420818 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:07.420822 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:07.420831 (+ 9us) server_negotiation.cc:407] Connection header received
0620 09:30:07.420868 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:07.420870 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:07.420884 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:07.420952 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:30:07.421214 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:07.422285 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:07.424761 (+ 2476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:07.425558 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:07.425640 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:07.425739 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:07.425929 (+ 190us) server_negotiation.cc:299] Negotiation successful
0620 09:30:07.425969 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":90,"threads_started":1}
I20260620 09:30:07.426106 4172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4fea0b97f8e947978bc57b749f2a4774. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:30:07.426059 3910 tablet_service.cc:1467] Processing CreateTablet for tablet 4fea0b97f8e947978bc57b749f2a4774 (DEFAULT_TABLE table=test-table [id=fc65cb4dfe844f6dab9e13407bc297d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:30:07.426391 3910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4fea0b97f8e947978bc57b749f2a4774. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:30:07.426584 4041 tablet_service.cc:1467] Processing CreateTablet for tablet 4fea0b97f8e947978bc57b749f2a4774 (DEFAULT_TABLE table=test-table [id=fc65cb4dfe844f6dab9e13407bc297d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:30:07.426826 4041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4fea0b97f8e947978bc57b749f2a4774. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:30:07.429454 4252 tablet_bootstrap.cc:492] T 4fea0b97f8e947978bc57b749f2a4774 P 9223be19b5b64d089215a99f669e2f40: Bootstrap starting.
I20260620 09:30:07.430163 4252 tablet_bootstrap.cc:654] T 4fea0b97f8e947978bc57b749f2a4774 P 9223be19b5b64d089215a99f669e2f40: Neither blocks nor log segments found. Creating new log.
I20260620 09:30:07.430475 4252 log.cc:824] T 4fea0b97f8e947978bc57b749f2a4774 P 9223be19b5b64d089215a99f669e2f40: Log is configured to *not* fsync() on all Append() calls
I20260620 09:30:07.430846 4254 tablet_bootstrap.cc:492] T 4fea0b97f8e947978bc57b749f2a4774 P c168e12565444f959606259299318f32: Bootstrap starting.
I20260620 09:30:07.431277 4252 tablet_bootstrap.cc:492] T 4fea0b97f8e947978bc57b749f2a4774 P 9223be19b5b64d089215a99f669e2f40: No bootstrap required, opened a new log
I20260620 09:30:07.431368 4252 ts_tablet_manager.cc:1397] T 4fea0b97f8e947978bc57b749f2a4774 P 9223be19b5b64d089215a99f669e2f40: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 09:30:07.431455 4254 tablet_bootstrap.cc:654] T 4fea0b97f8e947978bc57b749f2a4774 P c168e12565444f959606259299318f32: Neither blocks nor log segments found. Creating new log.
I20260620 09:30:07.431771 4254 log.cc:824] T 4fea0b97f8e947978bc57b749f2a4774 P c168e12565444f959606259299318f32: Log is configured to *not* fsync() on all Append() calls
I20260620 09:30:07.432418 4254 tablet_bootstrap.cc:492] T 4fea0b97f8e947978bc57b749f2a4774 P c168e12565444f959606259299318f32: No bootstrap required, opened a new log
I20260620 09:30:07.432471 4254 ts_tablet_manager.cc:1397] T 4fea0b97f8e947978bc57b749f2a4774 P c168e12565444f959606259299318f32: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 09:30:07.433727 4252 raft_consensus.cc:348] T 4fea0b97f8e947978bc57b749f2a4774 P 9223be19b5b64d089215a99f669e2f40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c168e12565444f959606259299318f32" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 35391 } } peers { permanent_uuid: "aa49757dd7d54a9fbe9c3f1ddcba70b0" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 44289 } } peers { permanent_uuid: "9223be19b5b64d089215a99f669e2f40" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 43723 } }
I20260620 09:30:07.433892 4252 raft_consensus.cc:374] T 4fea0b97f8e947978bc57b749f2a4774 P 9223be19b5b64d089215a99f669e2f40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:30:07.433939 4252 raft_consensus.cc:729] T 4fea0b97f8e947978bc57b749f2a4774 P 9223be19b5b64d089215a99f669e2f40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9223be19b5b64d089215a99f669e2f40, State: Initialized, Role: FOLLOWER
I20260620 09:30:07.434067 4252 consensus_queue.cc:260] T 4fea0b97f8e947978bc57b749f2a4774 P 9223be19b5b64d089215a99f669e2f40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c168e12565444f959606259299318f32" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 35391 } } peers { permanent_uuid: "aa49757dd7d54a9fbe9c3f1ddcba70b0" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 44289 } } peers { permanent_uuid: "9223be19b5b64d089215a99f669e2f40" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 43723 } }
I20260620 09:30:07.434340 4252 ts_tablet_manager.cc:1428] T 4fea0b97f8e947978bc57b749f2a4774 P 9223be19b5b64d089215a99f669e2f40: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 09:30:07.434417 4107 heartbeater.cc:503] Master 127.16.43.190:45093 was elected leader, sending a full tablet report...
I20260620 09:30:07.434396 4254 raft_consensus.cc:348] T 4fea0b97f8e947978bc57b749f2a4774 P c168e12565444f959606259299318f32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c168e12565444f959606259299318f32" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 35391 } } peers { permanent_uuid: "aa49757dd7d54a9fbe9c3f1ddcba70b0" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 44289 } } peers { permanent_uuid: "9223be19b5b64d089215a99f669e2f40" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 43723 } }
I20260620 09:30:07.434556 4254 raft_consensus.cc:374] T 4fea0b97f8e947978bc57b749f2a4774 P c168e12565444f959606259299318f32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:30:07.434588 4254 raft_consensus.cc:729] T 4fea0b97f8e947978bc57b749f2a4774 P c168e12565444f959606259299318f32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c168e12565444f959606259299318f32, State: Initialized, Role: FOLLOWER
I20260620 09:30:07.434695 4254 consensus_queue.cc:260] T 4fea0b97f8e947978bc57b749f2a4774 P c168e12565444f959606259299318f32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c168e12565444f959606259299318f32" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 35391 } } peers { permanent_uuid: "aa49757dd7d54a9fbe9c3f1ddcba70b0" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 44289 } } peers { permanent_uuid: "9223be19b5b64d089215a99f669e2f40" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 43723 } }
I20260620 09:30:07.434974 4254 ts_tablet_manager.cc:1428] T 4fea0b97f8e947978bc57b749f2a4774 P c168e12565444f959606259299318f32: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260620 09:30:07.432102 4253 tablet_bootstrap.cc:492] T 4fea0b97f8e947978bc57b749f2a4774 P aa49757dd7d54a9fbe9c3f1ddcba70b0: Bootstrap starting.
I20260620 09:30:07.435030 3976 heartbeater.cc:503] Master 127.16.43.190:45093 was elected leader, sending a full tablet report...
I20260620 09:30:07.435770 4253 tablet_bootstrap.cc:654] T 4fea0b97f8e947978bc57b749f2a4774 P aa49757dd7d54a9fbe9c3f1ddcba70b0: Neither blocks nor log segments found. Creating new log.
I20260620 09:30:07.436061 4253 log.cc:824] T 4fea0b97f8e947978bc57b749f2a4774 P aa49757dd7d54a9fbe9c3f1ddcba70b0: Log is configured to *not* fsync() on all Append() calls
I20260620 09:30:07.436928 4253 tablet_bootstrap.cc:492] T 4fea0b97f8e947978bc57b749f2a4774 P aa49757dd7d54a9fbe9c3f1ddcba70b0: No bootstrap required, opened a new log
I20260620 09:30:07.437008 4253 ts_tablet_manager.cc:1397] T 4fea0b97f8e947978bc57b749f2a4774 P aa49757dd7d54a9fbe9c3f1ddcba70b0: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260620 09:30:07.438899 4253 raft_consensus.cc:348] T 4fea0b97f8e947978bc57b749f2a4774 P aa49757dd7d54a9fbe9c3f1ddcba70b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c168e12565444f959606259299318f32" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 35391 } } peers { permanent_uuid: "aa49757dd7d54a9fbe9c3f1ddcba70b0" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 44289 } } peers { permanent_uuid: "9223be19b5b64d089215a99f669e2f40" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 43723 } }
I20260620 09:30:07.439064 4253 raft_consensus.cc:374] T 4fea0b97f8e947978bc57b749f2a4774 P aa49757dd7d54a9fbe9c3f1ddcba70b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:30:07.439111 4253 raft_consensus.cc:729] T 4fea0b97f8e947978bc57b749f2a4774 P aa49757dd7d54a9fbe9c3f1ddcba70b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa49757dd7d54a9fbe9c3f1ddcba70b0, State: Initialized, Role: FOLLOWER
I20260620 09:30:07.439260 4253 consensus_queue.cc:260] T 4fea0b97f8e947978bc57b749f2a4774 P aa49757dd7d54a9fbe9c3f1ddcba70b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c168e12565444f959606259299318f32" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 35391 } } peers { permanent_uuid: "aa49757dd7d54a9fbe9c3f1ddcba70b0" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 44289 } } peers { permanent_uuid: "9223be19b5b64d089215a99f669e2f40" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 43723 } }
I20260620 09:30:07.439564 4253 ts_tablet_manager.cc:1428] T 4fea0b97f8e947978bc57b749f2a4774 P aa49757dd7d54a9fbe9c3f1ddcba70b0: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 09:30:07.439651 4238 heartbeater.cc:503] Master 127.16.43.190:45093 was elected leader, sending a full tablet report...
I20260620 09:30:07.458482 4257 raft_consensus.cc:482] T 4fea0b97f8e947978bc57b749f2a4774 P 9223be19b5b64d089215a99f669e2f40 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:30:07.458631 4257 raft_consensus.cc:504] T 4fea0b97f8e947978bc57b749f2a4774 P 9223be19b5b64d089215a99f669e2f40 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c168e12565444f959606259299318f32" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 35391 } } peers { permanent_uuid: "aa49757dd7d54a9fbe9c3f1ddcba70b0" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 44289 } } peers { permanent_uuid: "9223be19b5b64d089215a99f669e2f40" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 43723 } }
I20260620 09:30:07.459187 4257 leader_election.cc:288] T 4fea0b97f8e947978bc57b749f2a4774 P 9223be19b5b64d089215a99f669e2f40 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c168e12565444f959606259299318f32 (127.16.43.129:35391), aa49757dd7d54a9fbe9c3f1ddcba70b0 (127.16.43.131:44289)
I20260620 09:30:07.462316 4251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:07.459489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.130:40567 (local address 127.16.43.129:35391)
0620 09:30:07.459558 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:07.459560 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:07.459569 (+ 9us) server_negotiation.cc:407] Connection header received
0620 09:30:07.459596 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:07.459597 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:07.459614 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:07.459681 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:30:07.459876 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:07.460365 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:07.461549 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:07.461927 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:07.462021 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:07.462030 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:07.462182 (+ 152us) server_negotiation.cc:299] Negotiation successful
0620 09:30:07.462223 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 09:30:07.462327 4109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:07.459358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.129:35391 (local address 127.16.43.130:40567)
0620 09:30:07.459422 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0620 09:30:07.459436 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 09:30:07.459488 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:30:07.459687 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:30:07.459688 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:30:07.459690 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:30:07.459799 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:30:07.459801 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:30:07.460460 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:30:07.460461 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:30:07.461461 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:30:07.461464 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:30:07.462021 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:30:07.462022 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:30:07.462093 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:07.462097 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 09:30:07.462173 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 09:30:07.462222 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260620 09:30:07.462337 4249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:07.459441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.43.130:36411 (local address 127.16.43.131:44289)
0620 09:30:07.459492 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:07.459494 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:07.459678 (+ 184us) server_negotiation.cc:407] Connection header received
0620 09:30:07.459778 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:07.459783 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:07.459792 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:07.459853 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:30:07.460004 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:07.460459 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:07.461607 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:07.461976 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:07.462059 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:07.462067 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:07.462217 (+ 150us) server_negotiation.cc:299] Negotiation successful
0620 09:30:07.462259 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 09:30:07.462460 4261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:07.459393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.131:44289 (local address 127.16.43.130:36411)
0620 09:30:07.459599 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0620 09:30:07.459609 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 09:30:07.459699 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:30:07.459856 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:30:07.459857 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:30:07.459858 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:30:07.459927 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:30:07.459928 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:30:07.460543 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:30:07.460544 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:30:07.461517 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:30:07.461520 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:30:07.462060 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:30:07.462061 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:30:07.462142 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:07.462150 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 09:30:07.462212 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 09:30:07.462257 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":108,"threads_started":1}
I20260620 09:30:07.462711 3930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fea0b97f8e947978bc57b749f2a4774" candidate_uuid: "9223be19b5b64d089215a99f669e2f40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c168e12565444f959606259299318f32" is_pre_election: true
I20260620 09:30:07.462792 4192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fea0b97f8e947978bc57b749f2a4774" candidate_uuid: "9223be19b5b64d089215a99f669e2f40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa49757dd7d54a9fbe9c3f1ddcba70b0" is_pre_election: true
I20260620 09:30:07.462919 4192 raft_consensus.cc:2454] T 4fea0b97f8e947978bc57b749f2a4774 P aa49757dd7d54a9fbe9c3f1ddcba70b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9223be19b5b64d089215a99f669e2f40 in term 0.
I20260620 09:30:07.462910 3930 raft_consensus.cc:2454] T 4fea0b97f8e947978bc57b749f2a4774 P c168e12565444f959606259299318f32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9223be19b5b64d089215a99f669e2f40 in term 0.
I20260620 09:30:07.463138 3996 leader_election.cc:302] T 4fea0b97f8e947978bc57b749f2a4774 P 9223be19b5b64d089215a99f669e2f40 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9223be19b5b64d089215a99f669e2f40, aa49757dd7d54a9fbe9c3f1ddcba70b0; no voters:
I20260620 09:30:07.463259 4257 raft_consensus.cc:2781] T 4fea0b97f8e947978bc57b749f2a4774 P 9223be19b5b64d089215a99f669e2f40 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 09:30:07.463322 4257 raft_consensus.cc:482] T 4fea0b97f8e947978bc57b749f2a4774 P 9223be19b5b64d089215a99f669e2f40 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 09:30:07.463352 4257 raft_consensus.cc:3037] T 4fea0b97f8e947978bc57b749f2a4774 P 9223be19b5b64d089215a99f669e2f40 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:30:07.464154 4257 raft_consensus.cc:504] T 4fea0b97f8e947978bc57b749f2a4774 P 9223be19b5b64d089215a99f669e2f40 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c168e12565444f959606259299318f32" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 35391 } } peers { permanent_uuid: "aa49757dd7d54a9fbe9c3f1ddcba70b0" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 44289 } } peers { permanent_uuid: "9223be19b5b64d089215a99f669e2f40" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 43723 } }
I20260620 09:30:07.464326 4257 leader_election.cc:288] T 4fea0b97f8e947978bc57b749f2a4774 P 9223be19b5b64d089215a99f669e2f40 [CANDIDATE]: Term 1 election: Requested vote from peers c168e12565444f959606259299318f32 (127.16.43.129:35391), aa49757dd7d54a9fbe9c3f1ddcba70b0 (127.16.43.131:44289)
I20260620 09:30:07.464497 4192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fea0b97f8e947978bc57b749f2a4774" candidate_uuid: "9223be19b5b64d089215a99f669e2f40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa49757dd7d54a9fbe9c3f1ddcba70b0"
I20260620 09:30:07.464552 3930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fea0b97f8e947978bc57b749f2a4774" candidate_uuid: "9223be19b5b64d089215a99f669e2f40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c168e12565444f959606259299318f32"
I20260620 09:30:07.464608 4192 raft_consensus.cc:3037] T 4fea0b97f8e947978bc57b749f2a4774 P aa49757dd7d54a9fbe9c3f1ddcba70b0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:30:07.464654 3930 raft_consensus.cc:3037] T 4fea0b97f8e947978bc57b749f2a4774 P c168e12565444f959606259299318f32 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:30:07.465426 4192 raft_consensus.cc:2454] T 4fea0b97f8e947978bc57b749f2a4774 P aa49757dd7d54a9fbe9c3f1ddcba70b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9223be19b5b64d089215a99f669e2f40 in term 1.
I20260620 09:30:07.465597 3996 leader_election.cc:302] T 4fea0b97f8e947978bc57b749f2a4774 P 9223be19b5b64d089215a99f669e2f40 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9223be19b5b64d089215a99f669e2f40, aa49757dd7d54a9fbe9c3f1ddcba70b0; no voters:
I20260620 09:30:07.465637 3930 raft_consensus.cc:2454] T 4fea0b97f8e947978bc57b749f2a4774 P c168e12565444f959606259299318f32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9223be19b5b64d089215a99f669e2f40 in term 1.
I20260620 09:30:07.465688 4257 raft_consensus.cc:2781] T 4fea0b97f8e947978bc57b749f2a4774 P 9223be19b5b64d089215a99f669e2f40 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:30:07.465849 4257 raft_consensus.cc:686] T 4fea0b97f8e947978bc57b749f2a4774 P 9223be19b5b64d089215a99f669e2f40 [term 1 LEADER]: Becoming Leader. State: Replica: 9223be19b5b64d089215a99f669e2f40, State: Running, Role: LEADER
I20260620 09:30:07.465950 4257 consensus_queue.cc:237] T 4fea0b97f8e947978bc57b749f2a4774 P 9223be19b5b64d089215a99f669e2f40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c168e12565444f959606259299318f32" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 35391 } } peers { permanent_uuid: "aa49757dd7d54a9fbe9c3f1ddcba70b0" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 44289 } } peers { permanent_uuid: "9223be19b5b64d089215a99f669e2f40" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 43723 } }
I20260620 09:30:07.466873 3787 catalog_manager.cc:5495] T 4fea0b97f8e947978bc57b749f2a4774 P 9223be19b5b64d089215a99f669e2f40 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9223be19b5b64d089215a99f669e2f40 (127.16.43.130). New cstate: current_term: 1 leader_uuid: "9223be19b5b64d089215a99f669e2f40" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c168e12565444f959606259299318f32" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 35391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aa49757dd7d54a9fbe9c3f1ddcba70b0" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 44289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9223be19b5b64d089215a99f669e2f40" member_type: VOTER last_known_addr { host: "127.16.43.130" port: 43723 } health_report { overall_health: HEALTHY } } }
I20260620 09:30:07.475615 4250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:07.473140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40702 (local address 127.16.43.130:43723)
0620 09:30:07.473210 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:07.473219 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:07.473241 (+ 22us) server_negotiation.cc:407] Connection header received
0620 09:30:07.473281 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:07.473282 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:07.473301 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:07.473375 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:30:07.473509 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:07.474008 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:07.474789 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:07.474986 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:07.475062 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:07.475238 (+ 176us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:30:07.475345 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:30:07.475439 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:07.475457 (+ 18us) server_negotiation.cc:299] Negotiation successful
0620 09:30:07.475479 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 09:30:07.477479 4192 raft_consensus.cc:1261] T 4fea0b97f8e947978bc57b749f2a4774 P aa49757dd7d54a9fbe9c3f1ddcba70b0 [term 1 FOLLOWER]: Refusing update from remote peer 9223be19b5b64d089215a99f669e2f40: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:30:07.477494 3930 raft_consensus.cc:1261] T 4fea0b97f8e947978bc57b749f2a4774 P c168e12565444f959606259299318f32 [term 1 FOLLOWER]: Refusing update from remote peer 9223be19b5b64d089215a99f669e2f40: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:30:07.477866 4257 consensus_queue.cc:1035] T 4fea0b97f8e947978bc57b749f2a4774 P 9223be19b5b64d089215a99f669e2f40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c168e12565444f959606259299318f32" member_type: VOTER last_known_addr { host: "127.16.43.129" port: 35391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:30:07.478016 4262 consensus_queue.cc:1035] T 4fea0b97f8e947978bc57b749f2a4774 P 9223be19b5b64d089215a99f669e2f40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa49757dd7d54a9fbe9c3f1ddcba70b0" member_type: VOTER last_known_addr { host: "127.16.43.131" port: 44289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:30:07.480677 4265 mvcc.cc:205] Tried to move back new op lower bound from 7298858219425222656 to 7298858219381366784. Current Snapshot: MvccSnapshot[applied={T|T < 7298858219425222656}]
I20260620 09:30:07.481451 4266 mvcc.cc:205] Tried to move back new op lower bound from 7298858219425222656 to 7298858219381366784. Current Snapshot: MvccSnapshot[applied={T|T < 7298858219425222656}]
I20260620 09:30:07.484002 3788 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39792:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:30:07.484108 3788 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39792:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:30:07.484926 3788 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5716b3c1340c4f2bba7c09ae276e52b4: Sending DeleteTablet for 3 replicas of tablet 4fea0b97f8e947978bc57b749f2a4774
I20260620 09:30:07.485306 4041 tablet_service.cc:1514] Processing DeleteTablet for tablet 4fea0b97f8e947978bc57b749f2a4774 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:30:07 UTC) from {username='slave'} at 127.0.0.1:40688
I20260620 09:30:07.485360 4172 tablet_service.cc:1514] Processing DeleteTablet for tablet 4fea0b97f8e947978bc57b749f2a4774 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:30:07 UTC) from {username='slave'} at 127.0.0.1:59432
I20260620 09:30:07.485546 3910 tablet_service.cc:1514] Processing DeleteTablet for tablet 4fea0b97f8e947978bc57b749f2a4774 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:30:07 UTC) from {username='slave'} at 127.0.0.1:44106
I20260620 09:30:07.485618 4275 tablet_replica.cc:330] T 4fea0b97f8e947978bc57b749f2a4774 P aa49757dd7d54a9fbe9c3f1ddcba70b0: stopping tablet replica
I20260620 09:30:07.485793 4275 raft_consensus.cc:2229] T 4fea0b97f8e947978bc57b749f2a4774 P aa49757dd7d54a9fbe9c3f1ddcba70b0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:30:07.485930 4275 raft_consensus.cc:2258] T 4fea0b97f8e947978bc57b749f2a4774 P aa49757dd7d54a9fbe9c3f1ddcba70b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:30:07.485924 4276 tablet_replica.cc:330] T 4fea0b97f8e947978bc57b749f2a4774 P c168e12565444f959606259299318f32: stopping tablet replica
I20260620 09:30:07.486095 4276 raft_consensus.cc:2229] T 4fea0b97f8e947978bc57b749f2a4774 P c168e12565444f959606259299318f32 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:30:07.486160 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 3848
I20260620 09:30:07.487077 4275 ts_tablet_manager.cc:1905] T 4fea0b97f8e947978bc57b749f2a4774 P aa49757dd7d54a9fbe9c3f1ddcba70b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:30:07.488368 4274 tablet_replica.cc:330] T 4fea0b97f8e947978bc57b749f2a4774 P 9223be19b5b64d089215a99f669e2f40: stopping tablet replica
I20260620 09:30:07.488468 4274 raft_consensus.cc:2229] T 4fea0b97f8e947978bc57b749f2a4774 P 9223be19b5b64d089215a99f669e2f40 [term 1 LEADER]: Raft consensus shutting down.
I20260620 09:30:07.488598 4274 raft_consensus.cc:2258] T 4fea0b97f8e947978bc57b749f2a4774 P 9223be19b5b64d089215a99f669e2f40 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:30:07.489112 4274 ts_tablet_manager.cc:1905] T 4fea0b97f8e947978bc57b749f2a4774 P 9223be19b5b64d089215a99f669e2f40: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:30:07.489136 4275 ts_tablet_manager.cc:1918] T 4fea0b97f8e947978bc57b749f2a4774 P aa49757dd7d54a9fbe9c3f1ddcba70b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:30:07.489212 4275 log.cc:1196] T 4fea0b97f8e947978bc57b749f2a4774 P aa49757dd7d54a9fbe9c3f1ddcba70b0: Deleting WAL directory at /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781947562949448-16558-0/minicluster-data/ts-2/wal/wals/4fea0b97f8e947978bc57b749f2a4774
I20260620 09:30:07.489478 4275 ts_tablet_manager.cc:1939] T 4fea0b97f8e947978bc57b749f2a4774 P aa49757dd7d54a9fbe9c3f1ddcba70b0: Deleting consensus metadata
I20260620 09:30:07.489936 3774 catalog_manager.cc:4841] TS aa49757dd7d54a9fbe9c3f1ddcba70b0 (127.16.43.131:44289): tablet 4fea0b97f8e947978bc57b749f2a4774 (table test-table [id=fc65cb4dfe844f6dab9e13407bc297d3]) successfully deleted
I20260620 09:30:07.490701 4274 ts_tablet_manager.cc:1918] T 4fea0b97f8e947978bc57b749f2a4774 P 9223be19b5b64d089215a99f669e2f40: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:30:07.490787 4274 log.cc:1196] T 4fea0b97f8e947978bc57b749f2a4774 P 9223be19b5b64d089215a99f669e2f40: Deleting WAL directory at /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781947562949448-16558-0/minicluster-data/ts-1/wal/wals/4fea0b97f8e947978bc57b749f2a4774
I20260620 09:30:07.491043 4274 ts_tablet_manager.cc:1939] T 4fea0b97f8e947978bc57b749f2a4774 P 9223be19b5b64d089215a99f669e2f40: Deleting consensus metadata
I20260620 09:30:07.491423 3772 catalog_manager.cc:4841] TS 9223be19b5b64d089215a99f669e2f40 (127.16.43.130:43723): tablet 4fea0b97f8e947978bc57b749f2a4774 (table test-table [id=fc65cb4dfe844f6dab9e13407bc297d3]) successfully deleted
W20260620 09:30:07.493737 3775 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.43.129:35391 (error 108)
I20260620 09:30:07.494309 4246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:07.494142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.129:35391 (local address 127.0.0.1:44112)
0620 09:30:07.494212 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0620 09:30:07.494260 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.129:35391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260620 09:30:07.494326 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 3979
W20260620 09:30:07.495486 3775 catalog_manager.cc:4584] TS c168e12565444f959606259299318f32 (127.16.43.129:35391): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4fea0b97f8e947978bc57b749f2a4774: Network error: Client connection negotiation failed: client connection to 127.16.43.129:35391: connect: Connection refused (error 111)
I20260620 09:30:07.499403 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 4110
I20260620 09:30:07.504408 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 3757
2026-06-20T09:30:07Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (728 ms)
[----------] 6 tests from EncryptionPolicyTest (8634 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781947562949448-16558-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:30:07 dist-test-slave-706q krb5kdc[4280](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:30:07 dist-test-slave-706q krb5kdc[4280](info): set up 2 sockets
Jun 20 09:30:07 dist-test-slave-706q krb5kdc[4280](info): commencing operation
krb5kdc: starting...
W20260620 09:30:08.552464 16558 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781947562949448-16558-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781947562949448-16558-0/krb5kdc/test-user.keytab.
Jun 20 09:30:08 dist-test-slave-706q krb5kdc[4280](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947808, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:30:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:30:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781947562949448-16558-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781947562949448-16558-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 09:30:08.675803 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781947562949448-16558-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781947562949448-16558-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781947562949448-16558-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44059
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781947562949448-16558-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44059
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781947562949448-16558-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781947562949448-16558-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781947562949448-16558-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:30:08.761752 4293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:30:08.761976 4293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:30:08.762002 4293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:30:08.763231 4293 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:30:08.763275 4293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:30:08.763290 4293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:30:08.763303 4293 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:30:08.763339 4293 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:30:08.764968 4293 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781947562949448-16558-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781947562949448-16558-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44059
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781947562949448-16558-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44059
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4293
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781947562949448-16558-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:30:08.765231 4293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:30:08.765478 4293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:30:08.770499 4299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:30:08.770493 4298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:30:08.770493 4301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:30:08.770917 4293 server_base.cc:1034] running on GCE node
I20260620 09:30:08.771165 4293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:30:08.771485 4293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:30:08.772668 4293 hybrid_clock.cc:648] HybridClock initialized: now 1781947808772674 us; error 60 us; skew 500 ppm
Jun 20 09:30:08 dist-test-slave-706q krb5kdc[4280](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947808, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:30:08.775434 4293 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 09:30:08.776028 4293 webserver.cc:460] Webserver started at http://127.0.0.1:36701/ using document root <none> and password file <none>
I20260620 09:30:08.776273 4293 fs_manager.cc:359] Metadata directory not provided
I20260620 09:30:08.776340 4293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:30:08.776485 4293 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:30:08.777549 4293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781947562949448-16558-0/minicluster-data/master-0/data/instance:
uuid: "7639d0e363e04367bc93ad42f9d31f8c"
format_stamp: "Formatted at 2026-06-20 09:30:08 on dist-test-slave-706q"
server_key: "540359f1e6d58b795fe3a21ffcee0518"
server_key_iv: "018671ef63b14e1264b371cba968907c"
server_key_version: "encryptionkey@0"
I20260620 09:30:08.777912 4293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781947562949448-16558-0/minicluster-data/master-0/wal/instance:
uuid: "7639d0e363e04367bc93ad42f9d31f8c"
format_stamp: "Formatted at 2026-06-20 09:30:08 on dist-test-slave-706q"
server_key: "540359f1e6d58b795fe3a21ffcee0518"
server_key_iv: "018671ef63b14e1264b371cba968907c"
server_key_version: "encryptionkey@0"
I20260620 09:30:08.779321 4293 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 09:30:08.780229 4308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:30:08.780400 4293 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 09:30:08.780495 4293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781947562949448-16558-0/minicluster-data/master-0/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781947562949448-16558-0/minicluster-data/master-0/wal
uuid: "7639d0e363e04367bc93ad42f9d31f8c"
format_stamp: "Formatted at 2026-06-20 09:30:08 on dist-test-slave-706q"
server_key: "540359f1e6d58b795fe3a21ffcee0518"
server_key_iv: "018671ef63b14e1264b371cba968907c"
server_key_version: "encryptionkey@0"
I20260620 09:30:08.780582 4293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781947562949448-16558-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781947562949448-16558-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781947562949448-16558-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:30:08.802135 4293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:30:08.804699 4293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:30:08.804869 4293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:30:08.809132 4293 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44059
I20260620 09:30:08.809190 4360 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44059 every 8 connection(s)
I20260620 09:30:08.809556 4293 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
I20260620 09:30:08.810452 4361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:30:08.811295 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 4293
I20260620 09:30:08.811403 16558 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781947562949448-16558-0/minicluster-data/master-0/wal/instance
I20260620 09:30:08.811642 16558 external_mini_cluster.cc:1428] Setting key 7e2973dbccffa15375c98835d6c42f32
I20260620 09:30:08.813159 4361 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7639d0e363e04367bc93ad42f9d31f8c: Bootstrap starting.
I20260620 09:30:08.813755 4361 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7639d0e363e04367bc93ad42f9d31f8c: Neither blocks nor log segments found. Creating new log.
I20260620 09:30:08.814047 4361 log.cc:824] T 00000000000000000000000000000000 P 7639d0e363e04367bc93ad42f9d31f8c: Log is configured to *not* fsync() on all Append() calls
I20260620 09:30:08.814796 4361 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7639d0e363e04367bc93ad42f9d31f8c: No bootstrap required, opened a new log
I20260620 09:30:08.816949 4361 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7639d0e363e04367bc93ad42f9d31f8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7639d0e363e04367bc93ad42f9d31f8c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44059 } }
I20260620 09:30:08.817134 4361 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7639d0e363e04367bc93ad42f9d31f8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:30:08.817214 4361 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7639d0e363e04367bc93ad42f9d31f8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7639d0e363e04367bc93ad42f9d31f8c, State: Initialized, Role: FOLLOWER
I20260620 09:30:08.817349 4361 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7639d0e363e04367bc93ad42f9d31f8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7639d0e363e04367bc93ad42f9d31f8c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44059 } }
I20260620 09:30:08.817435 4361 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7639d0e363e04367bc93ad42f9d31f8c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:30:08.817461 4361 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7639d0e363e04367bc93ad42f9d31f8c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:30:08.817525 4361 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7639d0e363e04367bc93ad42f9d31f8c [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:30:08.818145 4361 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7639d0e363e04367bc93ad42f9d31f8c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7639d0e363e04367bc93ad42f9d31f8c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44059 } }
I20260620 09:30:08.818267 4361 leader_election.cc:302] T 00000000000000000000000000000000 P 7639d0e363e04367bc93ad42f9d31f8c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7639d0e363e04367bc93ad42f9d31f8c; no voters:
I20260620 09:30:08.818432 4361 leader_election.cc:288] T 00000000000000000000000000000000 P 7639d0e363e04367bc93ad42f9d31f8c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:30:08.818485 4366 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7639d0e363e04367bc93ad42f9d31f8c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:30:08.818655 4366 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7639d0e363e04367bc93ad42f9d31f8c [term 1 LEADER]: Becoming Leader. State: Replica: 7639d0e363e04367bc93ad42f9d31f8c, State: Running, Role: LEADER
I20260620 09:30:08.818706 4361 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7639d0e363e04367bc93ad42f9d31f8c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:30:08.818763 4366 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7639d0e363e04367bc93ad42f9d31f8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7639d0e363e04367bc93ad42f9d31f8c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44059 } }
Jun 20 09:30:08 dist-test-slave-706q krb5kdc[4280](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947808, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 09:30:08.819109 4368 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7639d0e363e04367bc93ad42f9d31f8c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7639d0e363e04367bc93ad42f9d31f8c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7639d0e363e04367bc93ad42f9d31f8c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44059 } } }
I20260620 09:30:08.819101 4366 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7639d0e363e04367bc93ad42f9d31f8c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7639d0e363e04367bc93ad42f9d31f8c. Latest consensus state: current_term: 1 leader_uuid: "7639d0e363e04367bc93ad42f9d31f8c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7639d0e363e04367bc93ad42f9d31f8c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44059 } } }
I20260620 09:30:08.819209 4368 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7639d0e363e04367bc93ad42f9d31f8c [sys.catalog]: This master's current role is: LEADER
I20260620 09:30:08.819221 4366 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7639d0e363e04367bc93ad42f9d31f8c [sys.catalog]: This master's current role is: LEADER
I20260620 09:30:08.819444 4373 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:30:08.819958 4373 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:30:08.821496 4373 catalog_manager.cc:1347] Generated new cluster ID: 7a2788d47fe347449fe650b09aaf1f46
I20260620 09:30:08.821552 4373 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:30:08.823413 4364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:08.812261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50815 (local address 127.0.0.1:44059)
0620 09:30:08.812435 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:08.812446 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:08.812544 (+ 98us) server_negotiation.cc:407] Connection header received
0620 09:30:08.813099 (+ 555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:08.813105 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:08.813161 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:08.813325 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:30:08.813985 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:08.815144 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:08.816024 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:08.816432 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:08.816556 (+ 124us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:08.819469 (+ 2913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:30:08.819474 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:30:08.819474 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:30:08.819488 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:30:08.821730 (+ 2242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:30:08.822163 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:30:08.822167 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:30:08.822167 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:30:08.822289 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:30:08.822520 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:30:08.822521 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:30:08.822522 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:30:08.822833 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:30:08.822997 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:08.823116 (+ 119us) server_negotiation.cc:299] Negotiation successful
0620 09:30:08.823219 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":88,"threads_started":1}
I20260620 09:30:08.830552 16558 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 09:30:08.837920 4373 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:30:08.838389 4373 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:30:08.842226 4373 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7639d0e363e04367bc93ad42f9d31f8c: Generated new TSK 0
I20260620 09:30:08.842371 4373 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 09:30:08.843664 4364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:08.832116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42167 (local address 127.0.0.1:44059)
0620 09:30:08.832159 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:08.832161 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:08.835728 (+ 3567us) server_negotiation.cc:407] Connection header received
0620 09:30:08.835837 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:08.835838 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:08.835849 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:08.835878 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:30:08.836618 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:08.837417 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:08.838610 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:08.838947 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:08.838988 (+ 41us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:08.840614 (+ 1626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:30:08.840615 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:30:08.840616 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:30:08.840622 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:30:08.842209 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:30:08.842788 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:30:08.842788 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:30:08.842789 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:30:08.842846 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:30:08.843072 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:30:08.843073 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:30:08.843073 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:30:08.843173 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:30:08.843254 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:08.843441 (+ 187us) server_negotiation.cc:299] Negotiation successful
0620 09:30:08.843526 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 09:30:08.845614 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 4293
W20260620 09:30:08.849596 4278 connection.cc:535] client connection to 127.0.0.1:44059 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44059): Connection reset by peer (error 104)
2026-06-20T09:30:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1338 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781947562949448-16558-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:30:08 dist-test-slave-706q krb5kdc[4392](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:30:08 dist-test-slave-706q krb5kdc[4392](info): set up 2 sockets
Jun 20 09:30:08 dist-test-slave-706q krb5kdc[4392](info): commencing operation
krb5kdc: starting...
W20260620 09:30:09.884968 16558 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781947562949448-16558-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781947562949448-16558-0/krb5kdc/test-user.keytab.
Jun 20 09:30:09 dist-test-slave-706q krb5kdc[4392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947809, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:30:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:30:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781947562949448-16558-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781947562949448-16558-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 09:30:10.015120 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781947562949448-16558-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781947562949448-16558-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781947562949448-16558-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42767
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781947562949448-16558-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42767
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781947562949448-16558-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781947562949448-16558-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781947562949448-16558-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:30:10.099922 4405 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:30:10.100144 4405 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:30:10.100169 4405 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:30:10.101258 4405 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:30:10.101300 4405 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:30:10.101315 4405 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:30:10.101329 4405 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:30:10.101343 4405 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:30:10.102762 4405 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781947562949448-16558-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781947562949448-16558-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42767
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781947562949448-16558-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42767
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4405
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781947562949448-16558-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:30:10.102962 4405 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:30:10.103134 4405 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:30:10.108728 4413 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:30:10.108768 4410 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:30:10.108779 4405 server_base.cc:1034] running on GCE node
W20260620 09:30:10.108740 4411 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:30:10.109200 4405 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:30:10.109532 4405 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:30:10.110714 4405 hybrid_clock.cc:648] HybridClock initialized: now 1781947810110702 us; error 42 us; skew 500 ppm
Jun 20 09:30:10 dist-test-slave-706q krb5kdc[4392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947810, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:30:10.117005 4405 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 09:30:10.117528 4405 webserver.cc:460] Webserver started at http://127.0.0.1:44505/ using document root <none> and password file <none>
I20260620 09:30:10.117821 4405 fs_manager.cc:359] Metadata directory not provided
I20260620 09:30:10.117897 4405 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:30:10.118052 4405 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:30:10.119320 4405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781947562949448-16558-0/minicluster-data/master-0/data/instance:
uuid: "79ee06950ac244688ee43e7096d4c018"
format_stamp: "Formatted at 2026-06-20 09:30:10 on dist-test-slave-706q"
server_key: "3107f0d99565e81ab12a3b2bd77f52cd"
server_key_iv: "eef7fe8d28d0dda661ad337e1fda000b"
server_key_version: "encryptionkey@0"
I20260620 09:30:10.119768 4405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781947562949448-16558-0/minicluster-data/master-0/wal/instance:
uuid: "79ee06950ac244688ee43e7096d4c018"
format_stamp: "Formatted at 2026-06-20 09:30:10 on dist-test-slave-706q"
server_key: "3107f0d99565e81ab12a3b2bd77f52cd"
server_key_iv: "eef7fe8d28d0dda661ad337e1fda000b"
server_key_version: "encryptionkey@0"
I20260620 09:30:10.121340 4405 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 09:30:10.122264 4420 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:30:10.122609 4405 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 09:30:10.122730 4405 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781947562949448-16558-0/minicluster-data/master-0/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781947562949448-16558-0/minicluster-data/master-0/wal
uuid: "79ee06950ac244688ee43e7096d4c018"
format_stamp: "Formatted at 2026-06-20 09:30:10 on dist-test-slave-706q"
server_key: "3107f0d99565e81ab12a3b2bd77f52cd"
server_key_iv: "eef7fe8d28d0dda661ad337e1fda000b"
server_key_version: "encryptionkey@0"
I20260620 09:30:10.122824 4405 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781947562949448-16558-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781947562949448-16558-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781947562949448-16558-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:30:10.146663 4405 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:30:10.149106 4405 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:30:10.149281 4405 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:30:10.153623 4405 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42767
I20260620 09:30:10.153664 4472 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42767 every 8 connection(s)
I20260620 09:30:10.154022 4405 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
I20260620 09:30:10.154984 4473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:30:10.157433 4473 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79ee06950ac244688ee43e7096d4c018: Bootstrap starting.
I20260620 09:30:10.158073 4473 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 79ee06950ac244688ee43e7096d4c018: Neither blocks nor log segments found. Creating new log.
I20260620 09:30:10.158361 4473 log.cc:824] T 00000000000000000000000000000000 P 79ee06950ac244688ee43e7096d4c018: Log is configured to *not* fsync() on all Append() calls
I20260620 09:30:10.159178 4473 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79ee06950ac244688ee43e7096d4c018: No bootstrap required, opened a new log
I20260620 09:30:10.160841 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 4405
I20260620 09:30:10.160979 16558 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781947562949448-16558-0/minicluster-data/master-0/wal/instance
I20260620 09:30:10.161232 16558 external_mini_cluster.cc:1428] Setting key 1b2ddaf3bf4fc2309b001101fd5578e7
I20260620 09:30:10.161192 4473 raft_consensus.cc:348] T 00000000000000000000000000000000 P 79ee06950ac244688ee43e7096d4c018 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79ee06950ac244688ee43e7096d4c018" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42767 } }
I20260620 09:30:10.161382 4473 raft_consensus.cc:374] T 00000000000000000000000000000000 P 79ee06950ac244688ee43e7096d4c018 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:30:10.161440 4473 raft_consensus.cc:729] T 00000000000000000000000000000000 P 79ee06950ac244688ee43e7096d4c018 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79ee06950ac244688ee43e7096d4c018, State: Initialized, Role: FOLLOWER
I20260620 09:30:10.161616 4473 consensus_queue.cc:260] T 00000000000000000000000000000000 P 79ee06950ac244688ee43e7096d4c018 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79ee06950ac244688ee43e7096d4c018" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42767 } }
I20260620 09:30:10.161708 4473 raft_consensus.cc:388] T 00000000000000000000000000000000 P 79ee06950ac244688ee43e7096d4c018 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:30:10.161752 4473 raft_consensus.cc:482] T 00000000000000000000000000000000 P 79ee06950ac244688ee43e7096d4c018 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:30:10.161798 4473 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 79ee06950ac244688ee43e7096d4c018 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:30:10.162660 4473 raft_consensus.cc:504] T 00000000000000000000000000000000 P 79ee06950ac244688ee43e7096d4c018 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79ee06950ac244688ee43e7096d4c018" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42767 } }
I20260620 09:30:10.162753 4473 leader_election.cc:302] T 00000000000000000000000000000000 P 79ee06950ac244688ee43e7096d4c018 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 79ee06950ac244688ee43e7096d4c018; no voters:
I20260620 09:30:10.162906 4473 leader_election.cc:288] T 00000000000000000000000000000000 P 79ee06950ac244688ee43e7096d4c018 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:30:10.162963 4478 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 79ee06950ac244688ee43e7096d4c018 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:30:10.163110 4478 raft_consensus.cc:686] T 00000000000000000000000000000000 P 79ee06950ac244688ee43e7096d4c018 [term 1 LEADER]: Becoming Leader. State: Replica: 79ee06950ac244688ee43e7096d4c018, State: Running, Role: LEADER
I20260620 09:30:10.163175 4473 sys_catalog.cc:564] T 00000000000000000000000000000000 P 79ee06950ac244688ee43e7096d4c018 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:30:10.163206 4478 consensus_queue.cc:237] T 00000000000000000000000000000000 P 79ee06950ac244688ee43e7096d4c018 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79ee06950ac244688ee43e7096d4c018" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42767 } }
I20260620 09:30:10.163553 4479 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79ee06950ac244688ee43e7096d4c018 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "79ee06950ac244688ee43e7096d4c018" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79ee06950ac244688ee43e7096d4c018" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42767 } } }
I20260620 09:30:10.163651 4479 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79ee06950ac244688ee43e7096d4c018 [sys.catalog]: This master's current role is: LEADER
I20260620 09:30:10.163973 4480 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79ee06950ac244688ee43e7096d4c018 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 79ee06950ac244688ee43e7096d4c018. Latest consensus state: current_term: 1 leader_uuid: "79ee06950ac244688ee43e7096d4c018" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79ee06950ac244688ee43e7096d4c018" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42767 } } }
I20260620 09:30:10.164080 4480 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79ee06950ac244688ee43e7096d4c018 [sys.catalog]: This master's current role is: LEADER
I20260620 09:30:10.164376 4484 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:30:10.164911 4484 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:30:10.167423 4484 catalog_manager.cc:1347] Generated new cluster ID: 3f78934a55f046fe9624fab05c231de5
I20260620 09:30:10.167472 4484 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:30:10.167819 4477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:10.161800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47393 (local address 127.0.0.1:42767)
0620 09:30:10.162016 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:10.162028 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:10.162071 (+ 43us) server_negotiation.cc:407] Connection header received
0620 09:30:10.162217 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:10.162240 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:10.162296 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:10.162448 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:30:10.162679 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:10.164402 (+ 1723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:10.165614 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:10.166176 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:10.166324 (+ 148us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:10.167114 (+ 790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:30:10.167119 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:30:10.167119 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:30:10.167306 (+ 187us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:30:10.167368 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:30:10.167387 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:30:10.167477 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:10.167595 (+ 118us) server_negotiation.cc:299] Negotiation successful
0620 09:30:10.167624 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":98,"threads_started":1}
I20260620 09:30:10.172612 16558 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 09:30:10.174741 4484 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:30:10.175312 4484 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:30:10.178081 4477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:10.174124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56351 (local address 127.0.0.1:42767)
0620 09:30:10.174227 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:10.174230 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:10.174301 (+ 71us) server_negotiation.cc:407] Connection header received
0620 09:30:10.174417 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:10.174419 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:10.174430 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:10.174506 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:30:10.174772 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:10.175568 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:10.176646 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:10.176971 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:10.177046 (+ 75us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:10.177799 (+ 753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:30:10.177801 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:30:10.177801 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:30:10.177807 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:30:10.177822 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:30:10.177827 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:30:10.177887 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:10.177972 (+ 85us) server_negotiation.cc:299] Negotiation successful
0620 09:30:10.177986 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260620 09:30:10.181602 4484 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 79ee06950ac244688ee43e7096d4c018: Generated new TSK 0
I20260620 09:30:10.181819 4484 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 09:30:10.184921 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 4405
W20260620 09:30:10.188922 4390 connection.cc:535] client connection to 127.0.0.1:42767 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42767): Connection reset by peer (error 104)
2026-06-20T09:30:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1339 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781947562949448-16558-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:30:10 dist-test-slave-706q krb5kdc[4504](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:30:10 dist-test-slave-706q krb5kdc[4504](info): set up 2 sockets
Jun 20 09:30:10 dist-test-slave-706q krb5kdc[4504](info): commencing operation
krb5kdc: starting...
W20260620 09:30:11.224678 16558 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781947562949448-16558-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781947562949448-16558-0/krb5kdc/test-user.keytab.
Jun 20 09:30:11 dist-test-slave-706q krb5kdc[4504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947811, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:30:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:30:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781947562949448-16558-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781947562949448-16558-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 09:30:11.350662 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781947562949448-16558-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781947562949448-16558-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781947562949448-16558-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45599
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781947562949448-16558-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45599
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781947562949448-16558-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781947562949448-16558-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781947562949448-16558-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:30:11.434535 4517 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:30:11.434777 4517 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:30:11.434803 4517 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:30:11.435957 4517 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:30:11.436002 4517 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:30:11.436019 4517 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:30:11.436034 4517 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:30:11.436048 4517 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:30:11.437510 4517 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781947562949448-16558-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781947562949448-16558-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45599
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781947562949448-16558-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45599
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4517
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781947562949448-16558-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:30:11.437745 4517 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:30:11.437973 4517 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:30:11.443126 4523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:30:11.443105 4525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:30:11.443133 4517 server_base.cc:1034] running on GCE node
W20260620 09:30:11.443105 4522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:30:11.443691 4517 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:30:11.444000 4517 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:30:11.445147 4517 hybrid_clock.cc:648] HybridClock initialized: now 1781947811445128 us; error 42 us; skew 500 ppm
Jun 20 09:30:11 dist-test-slave-706q krb5kdc[4504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947811, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:30:11.447631 4517 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 09:30:11.448114 4517 webserver.cc:460] Webserver started at http://127.0.0.1:45029/ using document root <none> and password file <none>
I20260620 09:30:11.448338 4517 fs_manager.cc:359] Metadata directory not provided
I20260620 09:30:11.448412 4517 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:30:11.448562 4517 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:30:11.449626 4517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781947562949448-16558-0/minicluster-data/master-0/data/instance:
uuid: "5aab4322dadd4f7fb53f827f70b521aa"
format_stamp: "Formatted at 2026-06-20 09:30:11 on dist-test-slave-706q"
server_key: "e26bf42c9a3aa975ca835a830932bd01"
server_key_iv: "c84bbe9c7b076fd1c2b2001f5ca69f2e"
server_key_version: "encryptionkey@0"
I20260620 09:30:11.450001 4517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781947562949448-16558-0/minicluster-data/master-0/wal/instance:
uuid: "5aab4322dadd4f7fb53f827f70b521aa"
format_stamp: "Formatted at 2026-06-20 09:30:11 on dist-test-slave-706q"
server_key: "e26bf42c9a3aa975ca835a830932bd01"
server_key_iv: "c84bbe9c7b076fd1c2b2001f5ca69f2e"
server_key_version: "encryptionkey@0"
I20260620 09:30:11.451321 4517 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 09:30:11.452136 4532 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:30:11.452297 4517 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 09:30:11.452396 4517 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781947562949448-16558-0/minicluster-data/master-0/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781947562949448-16558-0/minicluster-data/master-0/wal
uuid: "5aab4322dadd4f7fb53f827f70b521aa"
format_stamp: "Formatted at 2026-06-20 09:30:11 on dist-test-slave-706q"
server_key: "e26bf42c9a3aa975ca835a830932bd01"
server_key_iv: "c84bbe9c7b076fd1c2b2001f5ca69f2e"
server_key_version: "encryptionkey@0"
I20260620 09:30:11.452479 4517 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781947562949448-16558-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781947562949448-16558-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781947562949448-16558-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:30:11.455787 4517 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:30:11.457770 4517 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:30:11.457931 4517 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:30:11.462009 4517 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45599
I20260620 09:30:11.462039 4584 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45599 every 8 connection(s)
I20260620 09:30:11.462388 4517 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
I20260620 09:30:11.463230 4585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:30:11.465521 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 4517
I20260620 09:30:11.465574 4585 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5aab4322dadd4f7fb53f827f70b521aa: Bootstrap starting.
I20260620 09:30:11.465647 16558 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781947562949448-16558-0/minicluster-data/master-0/wal/instance
I20260620 09:30:11.465875 16558 external_mini_cluster.cc:1428] Setting key c841de06b010835fe0a970a92318972b
I20260620 09:30:11.466264 4585 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5aab4322dadd4f7fb53f827f70b521aa: Neither blocks nor log segments found. Creating new log.
I20260620 09:30:11.466658 4585 log.cc:824] T 00000000000000000000000000000000 P 5aab4322dadd4f7fb53f827f70b521aa: Log is configured to *not* fsync() on all Append() calls
I20260620 09:30:11.467370 4585 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5aab4322dadd4f7fb53f827f70b521aa: No bootstrap required, opened a new log
I20260620 09:30:11.468858 4588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:11.466419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50355 (local address 127.0.0.1:45599)
0620 09:30:11.466691 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:11.466699 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:11.466729 (+ 30us) server_negotiation.cc:407] Connection header received
0620 09:30:11.467335 (+ 606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:11.467343 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:11.467382 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:11.467638 (+ 256us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:30:11.468295 (+ 657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:30:11.468298 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:30:11.468298 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:30:11.468302 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:30:11.468340 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:30:11.468351 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:30:11.468438 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:11.468510 (+ 72us) server_negotiation.cc:299] Negotiation successful
0620 09:30:11.468596 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":103,"threads_started":1}
I20260620 09:30:11.469818 4585 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5aab4322dadd4f7fb53f827f70b521aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aab4322dadd4f7fb53f827f70b521aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45599 } }
I20260620 09:30:11.470026 4585 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5aab4322dadd4f7fb53f827f70b521aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:30:11.470075 4585 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5aab4322dadd4f7fb53f827f70b521aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5aab4322dadd4f7fb53f827f70b521aa, State: Initialized, Role: FOLLOWER
I20260620 09:30:11.470063 16558 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 09:30:11.470196 4585 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5aab4322dadd4f7fb53f827f70b521aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aab4322dadd4f7fb53f827f70b521aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45599 } }
I20260620 09:30:11.470295 4585 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5aab4322dadd4f7fb53f827f70b521aa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:30:11.470338 4585 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5aab4322dadd4f7fb53f827f70b521aa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:30:11.470384 4585 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5aab4322dadd4f7fb53f827f70b521aa [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:30:11.470923 4585 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5aab4322dadd4f7fb53f827f70b521aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aab4322dadd4f7fb53f827f70b521aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45599 } }
I20260620 09:30:11.471042 4585 leader_election.cc:302] T 00000000000000000000000000000000 P 5aab4322dadd4f7fb53f827f70b521aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5aab4322dadd4f7fb53f827f70b521aa; no voters:
I20260620 09:30:11.471206 4585 leader_election.cc:288] T 00000000000000000000000000000000 P 5aab4322dadd4f7fb53f827f70b521aa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:30:11.471251 4594 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5aab4322dadd4f7fb53f827f70b521aa [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:30:11.471388 4594 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5aab4322dadd4f7fb53f827f70b521aa [term 1 LEADER]: Becoming Leader. State: Replica: 5aab4322dadd4f7fb53f827f70b521aa, State: Running, Role: LEADER
I20260620 09:30:11.471457 4585 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5aab4322dadd4f7fb53f827f70b521aa [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:30:11.471499 4594 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5aab4322dadd4f7fb53f827f70b521aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aab4322dadd4f7fb53f827f70b521aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45599 } }
I20260620 09:30:11.471939 4594 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5aab4322dadd4f7fb53f827f70b521aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5aab4322dadd4f7fb53f827f70b521aa. Latest consensus state: current_term: 1 leader_uuid: "5aab4322dadd4f7fb53f827f70b521aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aab4322dadd4f7fb53f827f70b521aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45599 } } }
I20260620 09:30:11.471956 4596 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5aab4322dadd4f7fb53f827f70b521aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5aab4322dadd4f7fb53f827f70b521aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aab4322dadd4f7fb53f827f70b521aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45599 } } }
I20260620 09:30:11.472098 4594 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5aab4322dadd4f7fb53f827f70b521aa [sys.catalog]: This master's current role is: LEADER
I20260620 09:30:11.472150 4596 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5aab4322dadd4f7fb53f827f70b521aa [sys.catalog]: This master's current role is: LEADER
I20260620 09:30:11.472430 4604 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:30:11.472906 4588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:11.471590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37799 (local address 127.0.0.1:45599)
0620 09:30:11.471636 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:11.471638 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:11.471755 (+ 117us) server_negotiation.cc:407] Connection header received
0620 09:30:11.471877 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:11.471878 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:11.471900 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:11.471961 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:30:11.472664 (+ 703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:30:11.472665 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:30:11.472666 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:30:11.472670 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:30:11.472684 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:30:11.472689 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:30:11.472719 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:11.472833 (+ 114us) server_negotiation.cc:299] Negotiation successful
0620 09:30:11.472847 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 09:30:11.473479 4604 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:30:11.474928 4604 catalog_manager.cc:1347] Generated new cluster ID: 1abb3165345340398b3292aaaff35c8a
I20260620 09:30:11.474980 4604 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:30:11.494062 4604 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:30:11.494278 4604 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:30:11.516394 4604 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5aab4322dadd4f7fb53f827f70b521aa: Generated new TSK 0
I20260620 09:30:11.516623 4604 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 09:30:11.528185 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 4517
2026-06-20T09:30:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1344 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781947562949448-16558-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:30:11 dist-test-slave-706q krb5kdc[4616](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:30:11 dist-test-slave-706q krb5kdc[4616](info): set up 2 sockets
Jun 20 09:30:11 dist-test-slave-706q krb5kdc[4616](info): commencing operation
krb5kdc: starting...
W20260620 09:30:12.576953 16558 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781947562949448-16558-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781947562949448-16558-0/krb5kdc/test-user.keytab.
Jun 20 09:30:12 dist-test-slave-706q krb5kdc[4616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947812, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:30:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:30:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781947562949448-16558-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781947562949448-16558-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 09:30:12.698132 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781947562949448-16558-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781947562949448-16558-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781947562949448-16558-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33431
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781947562949448-16558-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33431
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781947562949448-16558-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781947562949448-16558-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781947562949448-16558-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:30:12.780071 4629 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:30:12.780310 4629 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:30:12.780336 4629 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:30:12.781486 4629 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:30:12.781527 4629 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:30:12.781541 4629 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:30:12.781554 4629 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:30:12.781566 4629 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:30:12.783015 4629 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781947562949448-16558-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781947562949448-16558-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33431
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781947562949448-16558-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33431
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4629
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781947562949448-16558-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:30:12.783221 4629 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:30:12.783411 4629 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:30:12.788442 4637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:30:12.788470 4635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:30:12.788473 4634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:30:12.788731 4629 server_base.cc:1034] running on GCE node
I20260620 09:30:12.788972 4629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:30:12.789248 4629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:30:12.790421 4629 hybrid_clock.cc:648] HybridClock initialized: now 1781947812790411 us; error 40 us; skew 500 ppm
Jun 20 09:30:12 dist-test-slave-706q krb5kdc[4616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947812, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:30:12.792980 4629 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 09:30:12.793445 4629 webserver.cc:460] Webserver started at http://127.0.0.1:34959/ using document root <none> and password file <none>
I20260620 09:30:12.793653 4629 fs_manager.cc:359] Metadata directory not provided
I20260620 09:30:12.793694 4629 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:30:12.793781 4629 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:30:12.794803 4629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781947562949448-16558-0/minicluster-data/master-0/data/instance:
uuid: "814f66e84f4f42eabe070320d8705d6f"
format_stamp: "Formatted at 2026-06-20 09:30:12 on dist-test-slave-706q"
server_key: "b393fa3334ed94f6b888729913f826a2"
server_key_iv: "5ee06833b5aa0b620e464a6a2708b42b"
server_key_version: "encryptionkey@0"
I20260620 09:30:12.795102 4629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781947562949448-16558-0/minicluster-data/master-0/wal/instance:
uuid: "814f66e84f4f42eabe070320d8705d6f"
format_stamp: "Formatted at 2026-06-20 09:30:12 on dist-test-slave-706q"
server_key: "b393fa3334ed94f6b888729913f826a2"
server_key_iv: "5ee06833b5aa0b620e464a6a2708b42b"
server_key_version: "encryptionkey@0"
I20260620 09:30:12.796487 4629 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 09:30:12.797237 4644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:30:12.797421 4629 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 09:30:12.797518 4629 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781947562949448-16558-0/minicluster-data/master-0/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781947562949448-16558-0/minicluster-data/master-0/wal
uuid: "814f66e84f4f42eabe070320d8705d6f"
format_stamp: "Formatted at 2026-06-20 09:30:12 on dist-test-slave-706q"
server_key: "b393fa3334ed94f6b888729913f826a2"
server_key_iv: "5ee06833b5aa0b620e464a6a2708b42b"
server_key_version: "encryptionkey@0"
I20260620 09:30:12.797581 4629 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781947562949448-16558-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781947562949448-16558-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781947562949448-16558-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:30:12.817482 4629 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:30:12.819906 4629 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:30:12.820030 4629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:30:12.824090 4629 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33431
I20260620 09:30:12.824123 4696 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33431 every 8 connection(s)
I20260620 09:30:12.824491 4629 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
I20260620 09:30:12.825491 4697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:30:12.827977 4697 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 814f66e84f4f42eabe070320d8705d6f: Bootstrap starting.
I20260620 09:30:12.828578 4697 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 814f66e84f4f42eabe070320d8705d6f: Neither blocks nor log segments found. Creating new log.
I20260620 09:30:12.828873 4697 log.cc:824] T 00000000000000000000000000000000 P 814f66e84f4f42eabe070320d8705d6f: Log is configured to *not* fsync() on all Append() calls
I20260620 09:30:12.829602 4697 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 814f66e84f4f42eabe070320d8705d6f: No bootstrap required, opened a new log
I20260620 09:30:12.831244 4697 raft_consensus.cc:348] T 00000000000000000000000000000000 P 814f66e84f4f42eabe070320d8705d6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "814f66e84f4f42eabe070320d8705d6f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33431 } }
I20260620 09:30:12.831406 4697 raft_consensus.cc:374] T 00000000000000000000000000000000 P 814f66e84f4f42eabe070320d8705d6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:30:12.831462 4697 raft_consensus.cc:729] T 00000000000000000000000000000000 P 814f66e84f4f42eabe070320d8705d6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 814f66e84f4f42eabe070320d8705d6f, State: Initialized, Role: FOLLOWER
I20260620 09:30:12.831660 4697 consensus_queue.cc:260] T 00000000000000000000000000000000 P 814f66e84f4f42eabe070320d8705d6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "814f66e84f4f42eabe070320d8705d6f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33431 } }
I20260620 09:30:12.831763 4697 raft_consensus.cc:388] T 00000000000000000000000000000000 P 814f66e84f4f42eabe070320d8705d6f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:30:12.831805 4697 raft_consensus.cc:482] T 00000000000000000000000000000000 P 814f66e84f4f42eabe070320d8705d6f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:30:12.831863 4697 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 814f66e84f4f42eabe070320d8705d6f [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:30:12.832401 4697 raft_consensus.cc:504] T 00000000000000000000000000000000 P 814f66e84f4f42eabe070320d8705d6f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "814f66e84f4f42eabe070320d8705d6f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33431 } }
I20260620 09:30:12.832525 4697 leader_election.cc:302] T 00000000000000000000000000000000 P 814f66e84f4f42eabe070320d8705d6f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 814f66e84f4f42eabe070320d8705d6f; no voters:
I20260620 09:30:12.832713 4697 leader_election.cc:288] T 00000000000000000000000000000000 P 814f66e84f4f42eabe070320d8705d6f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:30:12.832758 4700 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 814f66e84f4f42eabe070320d8705d6f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:30:12.832981 4700 raft_consensus.cc:686] T 00000000000000000000000000000000 P 814f66e84f4f42eabe070320d8705d6f [term 1 LEADER]: Becoming Leader. State: Replica: 814f66e84f4f42eabe070320d8705d6f, State: Running, Role: LEADER
I20260620 09:30:12.832984 4697 sys_catalog.cc:564] T 00000000000000000000000000000000 P 814f66e84f4f42eabe070320d8705d6f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:30:12.833153 4700 consensus_queue.cc:237] T 00000000000000000000000000000000 P 814f66e84f4f42eabe070320d8705d6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "814f66e84f4f42eabe070320d8705d6f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33431 } }
I20260620 09:30:12.833364 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 4629
I20260620 09:30:12.833499 16558 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781947562949448-16558-0/minicluster-data/master-0/wal/instance
I20260620 09:30:12.833706 16558 external_mini_cluster.cc:1428] Setting key 99b9d0191ec7bedc92a258b339d20c88
I20260620 09:30:12.834179 4701 sys_catalog.cc:455] T 00000000000000000000000000000000 P 814f66e84f4f42eabe070320d8705d6f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "814f66e84f4f42eabe070320d8705d6f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "814f66e84f4f42eabe070320d8705d6f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33431 } } }
I20260620 09:30:12.834183 4702 sys_catalog.cc:455] T 00000000000000000000000000000000 P 814f66e84f4f42eabe070320d8705d6f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 814f66e84f4f42eabe070320d8705d6f. Latest consensus state: current_term: 1 leader_uuid: "814f66e84f4f42eabe070320d8705d6f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "814f66e84f4f42eabe070320d8705d6f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33431 } } }
I20260620 09:30:12.834280 4701 sys_catalog.cc:458] T 00000000000000000000000000000000 P 814f66e84f4f42eabe070320d8705d6f [sys.catalog]: This master's current role is: LEADER
I20260620 09:30:12.834302 4702 sys_catalog.cc:458] T 00000000000000000000000000000000 P 814f66e84f4f42eabe070320d8705d6f [sys.catalog]: This master's current role is: LEADER
I20260620 09:30:12.834724 4713 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:30:12.835376 4713 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:30:12.836861 4713 catalog_manager.cc:1347] Generated new cluster ID: 3b5d9adb2e18432d9163a2d6a7c9c29b
I20260620 09:30:12.836917 4713 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 09:30:12 dist-test-slave-706q krb5kdc[4616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947812, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 09:30:12.846287 4718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:12.834590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39073 (local address 127.0.0.1:33431)
0620 09:30:12.834820 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:12.834830 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:12.834862 (+ 32us) server_negotiation.cc:407] Connection header received
0620 09:30:12.834990 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:12.834997 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:12.835056 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:12.835202 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:30:12.835901 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:12.837332 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:12.838341 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:12.838977 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:12.839094 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:12.841988 (+ 2894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:30:12.841992 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:30:12.841993 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:30:12.842006 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:30:12.844556 (+ 2550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:30:12.844990 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:30:12.844993 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:30:12.844994 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:30:12.845113 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:30:12.845328 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:30:12.845329 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:30:12.845329 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:30:12.845646 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:30:12.845793 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:12.845925 (+ 132us) server_negotiation.cc:299] Negotiation successful
0620 09:30:12.846058 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":64,"threads_started":1}
I20260620 09:30:12.847417 16558 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 09:30:12.861312 4713 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:30:12.862102 4713 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:30:12.863377 4718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:12.848947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45365 (local address 127.0.0.1:33431)
0620 09:30:12.848985 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:12.848987 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:12.855775 (+ 6788us) server_negotiation.cc:407] Connection header received
0620 09:30:12.855849 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:12.855850 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:12.855864 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:12.855927 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:30:12.856633 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:12.857377 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:12.858527 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:12.858840 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:12.858896 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:12.860459 (+ 1563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:30:12.860460 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:30:12.860461 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:30:12.860468 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:30:12.862144 (+ 1676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:30:12.862537 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:30:12.862539 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:30:12.862539 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:30:12.862606 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:30:12.862802 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:30:12.862803 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:30:12.862804 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:30:12.862942 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:30:12.863058 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:12.863155 (+ 97us) server_negotiation.cc:299] Negotiation successful
0620 09:30:12.863259 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 09:30:12.870666 4713 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 814f66e84f4f42eabe070320d8705d6f: Generated new TSK 0
I20260620 09:30:12.870890 4713 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 09:30:12.872926 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 4629
2026-06-20T09:30:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1344 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781947562949448-16558-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:30:12 dist-test-slave-706q krb5kdc[4728](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:30:12 dist-test-slave-706q krb5kdc[4728](info): set up 2 sockets
Jun 20 09:30:12 dist-test-slave-706q krb5kdc[4728](info): commencing operation
krb5kdc: starting...
W20260620 09:30:13.960770 16558 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781947562949448-16558-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781947562949448-16558-0/krb5kdc/test-user.keytab.
Jun 20 09:30:14 dist-test-slave-706q krb5kdc[4728](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947814, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:30:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:30:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781947562949448-16558-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781947562949448-16558-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 09:30:14.087252 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781947562949448-16558-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781947562949448-16558-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781947562949448-16558-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33857
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781947562949448-16558-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33857
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781947562949448-16558-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781947562949448-16558-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781947562949448-16558-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:30:14.170049 4741 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:30:14.170285 4741 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:30:14.170310 4741 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:30:14.171487 4741 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:30:14.171617 4741 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:30:14.171661 4741 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:30:14.171677 4741 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:30:14.171689 4741 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:30:14.173249 4741 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781947562949448-16558-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781947562949448-16558-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33857
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781947562949448-16558-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33857
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4741
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781947562949448-16558-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:30:14.173454 4741 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:30:14.173635 4741 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:30:14.178583 4746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:30:14.178591 4747 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:30:14.178709 4741 server_base.cc:1034] running on GCE node
W20260620 09:30:14.178583 4749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:30:14.179088 4741 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:30:14.179404 4741 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:30:14.180543 4741 hybrid_clock.cc:648] HybridClock initialized: now 1781947814180531 us; error 32 us; skew 500 ppm
Jun 20 09:30:14 dist-test-slave-706q krb5kdc[4728](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947814, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:30:14.183028 4741 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 09:30:14.183569 4741 webserver.cc:460] Webserver started at http://127.0.0.1:46523/ using document root <none> and password file <none>
I20260620 09:30:14.183822 4741 fs_manager.cc:359] Metadata directory not provided
I20260620 09:30:14.183892 4741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:30:14.184036 4741 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:30:14.185138 4741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781947562949448-16558-0/minicluster-data/master-0/data/instance:
uuid: "21a87f1476c140c6997921d0554733f7"
format_stamp: "Formatted at 2026-06-20 09:30:14 on dist-test-slave-706q"
server_key: "fa78ca73c1f0cfd047101f147e453cb4"
server_key_iv: "001d09cb1356e1fff9bb59909dcada05"
server_key_version: "encryptionkey@0"
I20260620 09:30:14.185490 4741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781947562949448-16558-0/minicluster-data/master-0/wal/instance:
uuid: "21a87f1476c140c6997921d0554733f7"
format_stamp: "Formatted at 2026-06-20 09:30:14 on dist-test-slave-706q"
server_key: "fa78ca73c1f0cfd047101f147e453cb4"
server_key_iv: "001d09cb1356e1fff9bb59909dcada05"
server_key_version: "encryptionkey@0"
I20260620 09:30:14.186825 4741 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 09:30:14.187685 4756 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:30:14.187888 4741 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:30:14.187995 4741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781947562949448-16558-0/minicluster-data/master-0/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781947562949448-16558-0/minicluster-data/master-0/wal
uuid: "21a87f1476c140c6997921d0554733f7"
format_stamp: "Formatted at 2026-06-20 09:30:14 on dist-test-slave-706q"
server_key: "fa78ca73c1f0cfd047101f147e453cb4"
server_key_iv: "001d09cb1356e1fff9bb59909dcada05"
server_key_version: "encryptionkey@0"
I20260620 09:30:14.188077 4741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781947562949448-16558-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781947562949448-16558-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781947562949448-16558-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:30:14.211964 4741 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:30:14.214177 4741 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:30:14.214383 4741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:30:14.218689 4741 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33857
I20260620 09:30:14.218719 4808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33857 every 8 connection(s)
I20260620 09:30:14.219095 4741 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
I20260620 09:30:14.220080 4809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:30:14.222409 4809 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21a87f1476c140c6997921d0554733f7: Bootstrap starting.
I20260620 09:30:14.222788 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 4741
I20260620 09:30:14.222926 16558 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781947562949448-16558-0/minicluster-data/master-0/wal/instance
I20260620 09:30:14.223119 16558 external_mini_cluster.cc:1428] Setting key d052e059ebdae5fa6d3a353e546f169e
I20260620 09:30:14.223176 4809 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 21a87f1476c140c6997921d0554733f7: Neither blocks nor log segments found. Creating new log.
I20260620 09:30:14.223490 4809 log.cc:824] T 00000000000000000000000000000000 P 21a87f1476c140c6997921d0554733f7: Log is configured to *not* fsync() on all Append() calls
I20260620 09:30:14.224354 4809 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21a87f1476c140c6997921d0554733f7: No bootstrap required, opened a new log
I20260620 09:30:14.226794 4809 raft_consensus.cc:348] T 00000000000000000000000000000000 P 21a87f1476c140c6997921d0554733f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21a87f1476c140c6997921d0554733f7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33857 } }
I20260620 09:30:14.226974 4809 raft_consensus.cc:374] T 00000000000000000000000000000000 P 21a87f1476c140c6997921d0554733f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:30:14.227025 4809 raft_consensus.cc:729] T 00000000000000000000000000000000 P 21a87f1476c140c6997921d0554733f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21a87f1476c140c6997921d0554733f7, State: Initialized, Role: FOLLOWER
I20260620 09:30:14.227175 4809 consensus_queue.cc:260] T 00000000000000000000000000000000 P 21a87f1476c140c6997921d0554733f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21a87f1476c140c6997921d0554733f7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33857 } }
I20260620 09:30:14.227267 4809 raft_consensus.cc:388] T 00000000000000000000000000000000 P 21a87f1476c140c6997921d0554733f7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:30:14.227312 4809 raft_consensus.cc:482] T 00000000000000000000000000000000 P 21a87f1476c140c6997921d0554733f7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:30:14.227370 4809 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 21a87f1476c140c6997921d0554733f7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:30:14.228178 4809 raft_consensus.cc:504] T 00000000000000000000000000000000 P 21a87f1476c140c6997921d0554733f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21a87f1476c140c6997921d0554733f7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33857 } }
I20260620 09:30:14.228312 4809 leader_election.cc:302] T 00000000000000000000000000000000 P 21a87f1476c140c6997921d0554733f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 21a87f1476c140c6997921d0554733f7; no voters:
I20260620 09:30:14.228513 4809 leader_election.cc:288] T 00000000000000000000000000000000 P 21a87f1476c140c6997921d0554733f7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:30:14.228591 4814 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 21a87f1476c140c6997921d0554733f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:30:14.228782 4814 raft_consensus.cc:686] T 00000000000000000000000000000000 P 21a87f1476c140c6997921d0554733f7 [term 1 LEADER]: Becoming Leader. State: Replica: 21a87f1476c140c6997921d0554733f7, State: Running, Role: LEADER
I20260620 09:30:14.228859 4809 sys_catalog.cc:564] T 00000000000000000000000000000000 P 21a87f1476c140c6997921d0554733f7 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:30:14.228973 4814 consensus_queue.cc:237] T 00000000000000000000000000000000 P 21a87f1476c140c6997921d0554733f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21a87f1476c140c6997921d0554733f7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33857 } }
I20260620 09:30:14.229748 4813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:14.223645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34095 (local address 127.0.0.1:33857)
0620 09:30:14.223803 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:14.223820 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:14.223845 (+ 25us) server_negotiation.cc:407] Connection header received
0620 09:30:14.224443 (+ 598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:14.224452 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:14.224524 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:14.224681 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:30:14.224919 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:14.226337 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:14.227275 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:14.227928 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:14.228051 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:14.228947 (+ 896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:30:14.228952 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:30:14.228952 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:30:14.229160 (+ 208us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:30:14.229200 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:30:14.229214 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:30:14.229275 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:14.229418 (+ 143us) server_negotiation.cc:299] Negotiation successful
0620 09:30:14.229507 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":109,"thread_start_us":60,"threads_started":1}
I20260620 09:30:14.229965 4815 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21a87f1476c140c6997921d0554733f7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 21a87f1476c140c6997921d0554733f7. Latest consensus state: current_term: 1 leader_uuid: "21a87f1476c140c6997921d0554733f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21a87f1476c140c6997921d0554733f7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33857 } } }
I20260620 09:30:14.229987 4816 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21a87f1476c140c6997921d0554733f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "21a87f1476c140c6997921d0554733f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21a87f1476c140c6997921d0554733f7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33857 } } }
I20260620 09:30:14.230206 4816 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21a87f1476c140c6997921d0554733f7 [sys.catalog]: This master's current role is: LEADER
I20260620 09:30:14.230113 4815 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21a87f1476c140c6997921d0554733f7 [sys.catalog]: This master's current role is: LEADER
I20260620 09:30:14.230813 16558 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260620 09:30:14.231169 4829 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 21a87f1476c140c6997921d0554733f7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 09:30:14.231304 4829 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 09:30:14.231408 4826 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:30:14.231634 4826 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:30:14.233526 4826 catalog_manager.cc:1347] Generated new cluster ID: 05e6490efedd43728b98204cf1f3e941
I20260620 09:30:14.233577 4826 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:30:14.235816 4813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:14.232900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53339 (local address 127.0.0.1:33857)
0620 09:30:14.232944 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:14.232946 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:14.233069 (+ 123us) server_negotiation.cc:407] Connection header received
0620 09:30:14.233122 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:14.233123 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:14.233133 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:14.233203 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:30:14.233381 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:14.233871 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:14.234601 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:14.234804 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:14.234876 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:14.235571 (+ 695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:30:14.235573 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:30:14.235573 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:30:14.235579 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:30:14.235594 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:30:14.235598 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:30:14.235646 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:14.235711 (+ 65us) server_negotiation.cc:299] Negotiation successful
0620 09:30:14.235733 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 09:30:14.251943 4826 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:30:14.252529 4826 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:30:14.260114 4826 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 21a87f1476c140c6997921d0554733f7: Generated new TSK 0
I20260620 09:30:14.260322 4826 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 09:30:14.265652 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 4741
2026-06-20T09:30:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1392 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781947562949448-16558-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:30:14 dist-test-slave-706q krb5kdc[4840](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:30:14 dist-test-slave-706q krb5kdc[4840](info): set up 2 sockets
Jun 20 09:30:14 dist-test-slave-706q krb5kdc[4840](info): commencing operation
krb5kdc: starting...
W20260620 09:30:15.316871 16558 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781947562949448-16558-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781947562949448-16558-0/krb5kdc/test-user.keytab.
Jun 20 09:30:15 dist-test-slave-706q krb5kdc[4840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947815, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:30:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:30:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781947562949448-16558-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781947562949448-16558-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 09:30:15.442970 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781947562949448-16558-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781947562949448-16558-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781947562949448-16558-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37177
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781947562949448-16558-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37177
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781947562949448-16558-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781947562949448-16558-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781947562949448-16558-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:30:15.526043 4853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:30:15.526255 4853 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:30:15.526279 4853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:30:15.527374 4853 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:30:15.527410 4853 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:30:15.527424 4853 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:30:15.527437 4853 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:30:15.527449 4853 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:30:15.528988 4853 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781947562949448-16558-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781947562949448-16558-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37177
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781947562949448-16558-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37177
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4853
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781947562949448-16558-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:30:15.529244 4853 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:30:15.529495 4853 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:30:15.534691 4858 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:30:15.534791 4859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:30:15.534832 4853 server_base.cc:1034] running on GCE node
W20260620 09:30:15.534929 4861 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:30:15.535207 4853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:30:15.535507 4853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:30:15.536696 4853 hybrid_clock.cc:648] HybridClock initialized: now 1781947815536675 us; error 36 us; skew 500 ppm
Jun 20 09:30:15 dist-test-slave-706q krb5kdc[4840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947815, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:30:15.542438 4853 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 09:30:15.542915 4853 webserver.cc:460] Webserver started at http://127.0.0.1:37677/ using document root <none> and password file <none>
I20260620 09:30:15.543161 4853 fs_manager.cc:359] Metadata directory not provided
I20260620 09:30:15.543231 4853 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:30:15.543370 4853 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:30:15.544667 4853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781947562949448-16558-0/minicluster-data/master-0/data/instance:
uuid: "f371e24214494fb286db880853b46aef"
format_stamp: "Formatted at 2026-06-20 09:30:15 on dist-test-slave-706q"
server_key: "fda35c34561ed6ae638894e143980e8b"
server_key_iv: "1c5b03dd9b6ad64e33ac3144c02f5ec1"
server_key_version: "encryptionkey@0"
I20260620 09:30:15.545053 4853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781947562949448-16558-0/minicluster-data/master-0/wal/instance:
uuid: "f371e24214494fb286db880853b46aef"
format_stamp: "Formatted at 2026-06-20 09:30:15 on dist-test-slave-706q"
server_key: "fda35c34561ed6ae638894e143980e8b"
server_key_iv: "1c5b03dd9b6ad64e33ac3144c02f5ec1"
server_key_version: "encryptionkey@0"
I20260620 09:30:15.546430 4853 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 09:30:15.547227 4868 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:30:15.547395 4853 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 09:30:15.547499 4853 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781947562949448-16558-0/minicluster-data/master-0/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781947562949448-16558-0/minicluster-data/master-0/wal
uuid: "f371e24214494fb286db880853b46aef"
format_stamp: "Formatted at 2026-06-20 09:30:15 on dist-test-slave-706q"
server_key: "fda35c34561ed6ae638894e143980e8b"
server_key_iv: "1c5b03dd9b6ad64e33ac3144c02f5ec1"
server_key_version: "encryptionkey@0"
I20260620 09:30:15.547597 4853 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781947562949448-16558-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781947562949448-16558-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781947562949448-16558-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:30:15.550887 4853 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:30:15.553061 4853 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:30:15.553210 4853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:30:15.557176 4853 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37177
I20260620 09:30:15.557220 4920 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37177 every 8 connection(s)
I20260620 09:30:15.557550 4853 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
I20260620 09:30:15.558084 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 4853
I20260620 09:30:15.558223 16558 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781947562949448-16558-0/minicluster-data/master-0/wal/instance
I20260620 09:30:15.558434 16558 external_mini_cluster.cc:1428] Setting key d789761e7c34fc8449a2becb69b224a1
I20260620 09:30:15.558632 4921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:30:15.561160 4924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:15.558948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36787 (local address 127.0.0.1:37177)
0620 09:30:15.559132 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:15.559142 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:15.559174 (+ 32us) server_negotiation.cc:407] Connection header received
0620 09:30:15.559783 (+ 609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:15.559790 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:15.559816 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:15.559959 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:30:15.560694 (+ 735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:30:15.560698 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:30:15.560698 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:30:15.560703 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:30:15.560749 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:30:15.560759 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:30:15.560891 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:15.560962 (+ 71us) server_negotiation.cc:299] Negotiation successful
0620 09:30:15.561002 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":95,"threads_started":1}
I20260620 09:30:15.562026 4921 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f371e24214494fb286db880853b46aef: Bootstrap starting.
I20260620 09:30:15.562096 16558 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 09:30:15.562621 4921 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f371e24214494fb286db880853b46aef: Neither blocks nor log segments found. Creating new log.
I20260620 09:30:15.562916 4921 log.cc:824] T 00000000000000000000000000000000 P f371e24214494fb286db880853b46aef: Log is configured to *not* fsync() on all Append() calls
I20260620 09:30:15.563695 4921 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f371e24214494fb286db880853b46aef: No bootstrap required, opened a new log
I20260620 09:30:15.564584 4924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:15.563474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58789 (local address 127.0.0.1:37177)
0620 09:30:15.563513 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:15.563516 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:15.563646 (+ 130us) server_negotiation.cc:407] Connection header received
0620 09:30:15.563706 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:15.563706 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:15.563710 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:15.563776 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:30:15.564375 (+ 599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:30:15.564376 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:30:15.564377 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:30:15.564381 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:30:15.564395 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:30:15.564400 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:30:15.564456 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:15.564504 (+ 48us) server_negotiation.cc:299] Negotiation successful
0620 09:30:15.564518 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 09:30:15.565845 4921 raft_consensus.cc:348] T 00000000000000000000000000000000 P f371e24214494fb286db880853b46aef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f371e24214494fb286db880853b46aef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37177 } }
I20260620 09:30:15.565990 4921 raft_consensus.cc:374] T 00000000000000000000000000000000 P f371e24214494fb286db880853b46aef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:30:15.566016 4921 raft_consensus.cc:729] T 00000000000000000000000000000000 P f371e24214494fb286db880853b46aef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f371e24214494fb286db880853b46aef, State: Initialized, Role: FOLLOWER
I20260620 09:30:15.566126 4921 consensus_queue.cc:260] T 00000000000000000000000000000000 P f371e24214494fb286db880853b46aef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f371e24214494fb286db880853b46aef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37177 } }
I20260620 09:30:15.566222 4921 raft_consensus.cc:388] T 00000000000000000000000000000000 P f371e24214494fb286db880853b46aef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:30:15.566265 4921 raft_consensus.cc:482] T 00000000000000000000000000000000 P f371e24214494fb286db880853b46aef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:30:15.566322 4921 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f371e24214494fb286db880853b46aef [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:30:15.566854 4921 raft_consensus.cc:504] T 00000000000000000000000000000000 P f371e24214494fb286db880853b46aef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f371e24214494fb286db880853b46aef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37177 } }
I20260620 09:30:15.566943 4921 leader_election.cc:302] T 00000000000000000000000000000000 P f371e24214494fb286db880853b46aef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f371e24214494fb286db880853b46aef; no voters:
I20260620 09:30:15.567063 4921 leader_election.cc:288] T 00000000000000000000000000000000 P f371e24214494fb286db880853b46aef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:30:15.567134 4931 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f371e24214494fb286db880853b46aef [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:30:15.567291 4931 raft_consensus.cc:686] T 00000000000000000000000000000000 P f371e24214494fb286db880853b46aef [term 1 LEADER]: Becoming Leader. State: Replica: f371e24214494fb286db880853b46aef, State: Running, Role: LEADER
I20260620 09:30:15.567294 4921 sys_catalog.cc:564] T 00000000000000000000000000000000 P f371e24214494fb286db880853b46aef [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:30:15.567387 4931 consensus_queue.cc:237] T 00000000000000000000000000000000 P f371e24214494fb286db880853b46aef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f371e24214494fb286db880853b46aef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37177 } }
I20260620 09:30:15.567800 4933 sys_catalog.cc:455] T 00000000000000000000000000000000 P f371e24214494fb286db880853b46aef [sys.catalog]: SysCatalogTable state changed. Reason: New leader f371e24214494fb286db880853b46aef. Latest consensus state: current_term: 1 leader_uuid: "f371e24214494fb286db880853b46aef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f371e24214494fb286db880853b46aef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37177 } } }
I20260620 09:30:15.567930 4933 sys_catalog.cc:458] T 00000000000000000000000000000000 P f371e24214494fb286db880853b46aef [sys.catalog]: This master's current role is: LEADER
I20260620 09:30:15.567793 4932 sys_catalog.cc:455] T 00000000000000000000000000000000 P f371e24214494fb286db880853b46aef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f371e24214494fb286db880853b46aef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f371e24214494fb286db880853b46aef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37177 } } }
I20260620 09:30:15.568171 4938 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:30:15.568679 4938 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:30:15.568152 4932 sys_catalog.cc:458] T 00000000000000000000000000000000 P f371e24214494fb286db880853b46aef [sys.catalog]: This master's current role is: LEADER
I20260620 09:30:15.570225 4938 catalog_manager.cc:1347] Generated new cluster ID: 9d5d1623b09d4b35bee862d855bdce72
I20260620 09:30:15.570276 4938 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:30:15.589828 4938 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:30:15.590026 4938 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:30:15.608987 4938 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f371e24214494fb286db880853b46aef: Generated new TSK 0
I20260620 09:30:15.609225 4938 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 09:30:15.618474 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 4853
2026-06-20T09:30:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1352 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781947562949448-16558-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:30:15 dist-test-slave-706q krb5kdc[4952](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:30:15 dist-test-slave-706q krb5kdc[4952](info): set up 2 sockets
Jun 20 09:30:15 dist-test-slave-706q krb5kdc[4952](info): commencing operation
krb5kdc: starting...
W20260620 09:30:16.656692 16558 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781947562949448-16558-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781947562949448-16558-0/krb5kdc/test-user.keytab.
Jun 20 09:30:16 dist-test-slave-706q krb5kdc[4952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947816, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:30:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:30:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781947562949448-16558-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781947562949448-16558-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 09:30:16.783581 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781947562949448-16558-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781947562949448-16558-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781947562949448-16558-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33663
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781947562949448-16558-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33663
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781947562949448-16558-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781947562949448-16558-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781947562949448-16558-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:30:16.864763 4965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:30:16.865016 4965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:30:16.865054 4965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:30:16.866202 4965 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:30:16.866303 4965 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:30:16.866330 4965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:30:16.866356 4965 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:30:16.866389 4965 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:30:16.867859 4965 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781947562949448-16558-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781947562949448-16558-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33663
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781947562949448-16558-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33663
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4965
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781947562949448-16558-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:30:16.868129 4965 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:30:16.868367 4965 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:30:16.873152 4971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:30:16.873217 4973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:30:16.873392 4965 server_base.cc:1034] running on GCE node
W20260620 09:30:16.873245 4970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:30:16.873714 4965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:30:16.874058 4965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:30:16.875239 4965 hybrid_clock.cc:648] HybridClock initialized: now 1781947816875215 us; error 34 us; skew 500 ppm
Jun 20 09:30:16 dist-test-slave-706q krb5kdc[4952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947816, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:30:16.878021 4965 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 09:30:16.878495 4965 webserver.cc:460] Webserver started at http://127.0.0.1:34395/ using document root <none> and password file <none>
I20260620 09:30:16.878715 4965 fs_manager.cc:359] Metadata directory not provided
I20260620 09:30:16.878782 4965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:30:16.878942 4965 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:30:16.880108 4965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781947562949448-16558-0/minicluster-data/master-0/data/instance:
uuid: "e8ba94a0f1e447b18b193c9fc2dac6cf"
format_stamp: "Formatted at 2026-06-20 09:30:16 on dist-test-slave-706q"
server_key: "9cc942cc6eb30e288cefc29e1a470134"
server_key_iv: "8c2f61764a5bbf04e2663b9ef18615dd"
server_key_version: "encryptionkey@0"
I20260620 09:30:16.880470 4965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781947562949448-16558-0/minicluster-data/master-0/wal/instance:
uuid: "e8ba94a0f1e447b18b193c9fc2dac6cf"
format_stamp: "Formatted at 2026-06-20 09:30:16 on dist-test-slave-706q"
server_key: "9cc942cc6eb30e288cefc29e1a470134"
server_key_iv: "8c2f61764a5bbf04e2663b9ef18615dd"
server_key_version: "encryptionkey@0"
I20260620 09:30:16.881917 4965 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 09:30:16.882736 4980 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:30:16.882966 4965 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:30:16.883075 4965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781947562949448-16558-0/minicluster-data/master-0/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781947562949448-16558-0/minicluster-data/master-0/wal
uuid: "e8ba94a0f1e447b18b193c9fc2dac6cf"
format_stamp: "Formatted at 2026-06-20 09:30:16 on dist-test-slave-706q"
server_key: "9cc942cc6eb30e288cefc29e1a470134"
server_key_iv: "8c2f61764a5bbf04e2663b9ef18615dd"
server_key_version: "encryptionkey@0"
I20260620 09:30:16.883152 4965 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781947562949448-16558-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781947562949448-16558-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781947562949448-16558-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:30:16.905237 4965 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:30:16.907622 4965 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:30:16.907799 4965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:30:16.911890 4965 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33663
I20260620 09:30:16.911927 5032 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33663 every 8 connection(s)
I20260620 09:30:16.912283 4965 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
I20260620 09:30:16.913187 5033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:30:16.915462 5033 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8ba94a0f1e447b18b193c9fc2dac6cf: Bootstrap starting.
I20260620 09:30:16.916157 5033 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e8ba94a0f1e447b18b193c9fc2dac6cf: Neither blocks nor log segments found. Creating new log.
I20260620 09:30:16.916462 5033 log.cc:824] T 00000000000000000000000000000000 P e8ba94a0f1e447b18b193c9fc2dac6cf: Log is configured to *not* fsync() on all Append() calls
I20260620 09:30:16.917201 5033 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8ba94a0f1e447b18b193c9fc2dac6cf: No bootstrap required, opened a new log
I20260620 09:30:16.918706 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 4965
I20260620 09:30:16.918867 16558 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781947562949448-16558-0/minicluster-data/master-0/wal/instance
I20260620 09:30:16.919076 16558 external_mini_cluster.cc:1428] Setting key b6e368e644992402a6c5e8b4306d2b1e
I20260620 09:30:16.919201 5033 raft_consensus.cc:348] T 00000000000000000000000000000000 P e8ba94a0f1e447b18b193c9fc2dac6cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8ba94a0f1e447b18b193c9fc2dac6cf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33663 } }
I20260620 09:30:16.919395 5033 raft_consensus.cc:374] T 00000000000000000000000000000000 P e8ba94a0f1e447b18b193c9fc2dac6cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:30:16.919461 5033 raft_consensus.cc:729] T 00000000000000000000000000000000 P e8ba94a0f1e447b18b193c9fc2dac6cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8ba94a0f1e447b18b193c9fc2dac6cf, State: Initialized, Role: FOLLOWER
I20260620 09:30:16.919866 5033 consensus_queue.cc:260] T 00000000000000000000000000000000 P e8ba94a0f1e447b18b193c9fc2dac6cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8ba94a0f1e447b18b193c9fc2dac6cf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33663 } }
I20260620 09:30:16.919967 5033 raft_consensus.cc:388] T 00000000000000000000000000000000 P e8ba94a0f1e447b18b193c9fc2dac6cf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:30:16.920001 5033 raft_consensus.cc:482] T 00000000000000000000000000000000 P e8ba94a0f1e447b18b193c9fc2dac6cf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:30:16.920042 5033 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e8ba94a0f1e447b18b193c9fc2dac6cf [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:30:16.921121 5033 raft_consensus.cc:504] T 00000000000000000000000000000000 P e8ba94a0f1e447b18b193c9fc2dac6cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8ba94a0f1e447b18b193c9fc2dac6cf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33663 } }
I20260620 09:30:16.921229 5033 leader_election.cc:302] T 00000000000000000000000000000000 P e8ba94a0f1e447b18b193c9fc2dac6cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e8ba94a0f1e447b18b193c9fc2dac6cf; no voters:
I20260620 09:30:16.921403 5033 leader_election.cc:288] T 00000000000000000000000000000000 P e8ba94a0f1e447b18b193c9fc2dac6cf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:30:16.921490 5038 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e8ba94a0f1e447b18b193c9fc2dac6cf [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:30:16.921657 5033 sys_catalog.cc:564] T 00000000000000000000000000000000 P e8ba94a0f1e447b18b193c9fc2dac6cf [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:30:16.921666 5038 raft_consensus.cc:686] T 00000000000000000000000000000000 P e8ba94a0f1e447b18b193c9fc2dac6cf [term 1 LEADER]: Becoming Leader. State: Replica: e8ba94a0f1e447b18b193c9fc2dac6cf, State: Running, Role: LEADER
I20260620 09:30:16.921782 5038 consensus_queue.cc:237] T 00000000000000000000000000000000 P e8ba94a0f1e447b18b193c9fc2dac6cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8ba94a0f1e447b18b193c9fc2dac6cf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33663 } }
I20260620 09:30:16.922196 5038 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8ba94a0f1e447b18b193c9fc2dac6cf [sys.catalog]: SysCatalogTable state changed. Reason: New leader e8ba94a0f1e447b18b193c9fc2dac6cf. Latest consensus state: current_term: 1 leader_uuid: "e8ba94a0f1e447b18b193c9fc2dac6cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8ba94a0f1e447b18b193c9fc2dac6cf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33663 } } }
I20260620 09:30:16.922214 5039 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8ba94a0f1e447b18b193c9fc2dac6cf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e8ba94a0f1e447b18b193c9fc2dac6cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8ba94a0f1e447b18b193c9fc2dac6cf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33663 } } }
I20260620 09:30:16.922293 5038 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8ba94a0f1e447b18b193c9fc2dac6cf [sys.catalog]: This master's current role is: LEADER
I20260620 09:30:16.922295 5039 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8ba94a0f1e447b18b193c9fc2dac6cf [sys.catalog]: This master's current role is: LEADER
W20260620 09:30:16.923555 5053 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e8ba94a0f1e447b18b193c9fc2dac6cf: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 09:30:16.923655 5053 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 09:30:16.923753 5047 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:30:16.923959 5047 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:30:16.925664 5047 catalog_manager.cc:1347] Generated new cluster ID: b6f8697a709642509e4c72d68df81700
I20260620 09:30:16.925769 5047 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 09:30:16 dist-test-slave-706q krb5kdc[4952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947816, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 09:30:16.930064 5037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:16.919637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.41:43519 (local address 127.0.0.1:33663)
0620 09:30:16.919809 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:16.919819 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:16.919862 (+ 43us) server_negotiation.cc:407] Connection header received
0620 09:30:16.919983 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:16.919997 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:16.920059 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:16.920226 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:30:16.920809 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:16.922262 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:16.923449 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:16.923922 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:16.924058 (+ 136us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:16.926686 (+ 2628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:30:16.926690 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:30:16.926690 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:30:16.926704 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:30:16.928486 (+ 1782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:30:16.928901 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:30:16.928904 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:30:16.928904 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:30:16.929001 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:30:16.929236 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:30:16.929237 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:30:16.929238 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:30:16.929480 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:30:16.929671 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:16.929757 (+ 86us) server_negotiation.cc:299] Negotiation successful
0620 09:30:16.929875 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":93,"threads_started":1}
I20260620 09:30:16.931077 16558 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 09:30:16.936619 5047 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:30:16.938476 5047 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:30:16.939299 5037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:16.932527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.41:43357 (local address 127.0.0.1:33663)
0620 09:30:16.932567 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:16.932569 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:16.932676 (+ 107us) server_negotiation.cc:407] Connection header received
0620 09:30:16.932750 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:16.932751 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:16.932761 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:16.932819 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:30:16.933339 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:16.933840 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:16.934564 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:16.934790 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:16.934869 (+ 79us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:16.936357 (+ 1488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:30:16.936358 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:30:16.936358 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:30:16.936365 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:30:16.937950 (+ 1585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:30:16.938317 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:30:16.938318 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:30:16.938319 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:30:16.938395 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:30:16.938683 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:30:16.938684 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:30:16.938684 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:30:16.938793 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:30:16.938904 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:16.939113 (+ 209us) server_negotiation.cc:299] Negotiation successful
0620 09:30:16.939181 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 09:30:16.944967 5047 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e8ba94a0f1e447b18b193c9fc2dac6cf: Generated new TSK 0
I20260620 09:30:16.945178 5047 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 09:30:16.947075 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 4965
W20260620 09:30:16.951834 4950 connection.cc:535] client connection to 127.0.0.1:33663 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33663): Connection reset by peer (error 104)
2026-06-20T09:30:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1329 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781947562949448-16558-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:30:17 dist-test-slave-706q krb5kdc[5064](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:30:17 dist-test-slave-706q krb5kdc[5064](info): set up 2 sockets
Jun 20 09:30:17 dist-test-slave-706q krb5kdc[5064](info): commencing operation
krb5kdc: starting...
W20260620 09:30:17.996752 16558 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781947562949448-16558-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781947562949448-16558-0/krb5kdc/test-user.keytab.
Jun 20 09:30:18 dist-test-slave-706q krb5kdc[5064](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947818, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:30:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:30:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781947562949448-16558-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781947562949448-16558-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 09:30:18.121978 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781947562949448-16558-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781947562949448-16558-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781947562949448-16558-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36179
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781947562949448-16558-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36179
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781947562949448-16558-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781947562949448-16558-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781947562949448-16558-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:30:18.203131 5077 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:30:18.203367 5077 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:30:18.203392 5077 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:30:18.204674 5077 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:30:18.204728 5077 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:30:18.204744 5077 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:30:18.204757 5077 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:30:18.204802 5077 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:30:18.206360 5077 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781947562949448-16558-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781947562949448-16558-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36179
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781947562949448-16558-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36179
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5077
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781947562949448-16558-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:30:18.206625 5077 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:30:18.206876 5077 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:30:18.211835 5085 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:30:18.211947 5082 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:30:18.211835 5083 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:30:18.212037 5077 server_base.cc:1034] running on GCE node
I20260620 09:30:18.212354 5077 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:30:18.212643 5077 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:30:18.213914 5077 hybrid_clock.cc:648] HybridClock initialized: now 1781947818213891 us; error 52 us; skew 500 ppm
Jun 20 09:30:18 dist-test-slave-706q krb5kdc[5064](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947818, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:30:18.216542 5077 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 09:30:18.217000 5077 webserver.cc:460] Webserver started at http://127.0.0.1:42929/ using document root <none> and password file <none>
I20260620 09:30:18.217226 5077 fs_manager.cc:359] Metadata directory not provided
I20260620 09:30:18.217296 5077 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:30:18.217453 5077 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:30:18.218597 5077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781947562949448-16558-0/minicluster-data/master-0/data/instance:
uuid: "b7295a7448a148aa9364938832f805f9"
format_stamp: "Formatted at 2026-06-20 09:30:18 on dist-test-slave-706q"
server_key: "2352272790dadc90b756be6f6331a30b"
server_key_iv: "f207046b95b808e08c42457a1be6f33f"
server_key_version: "encryptionkey@0"
I20260620 09:30:18.218969 5077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781947562949448-16558-0/minicluster-data/master-0/wal/instance:
uuid: "b7295a7448a148aa9364938832f805f9"
format_stamp: "Formatted at 2026-06-20 09:30:18 on dist-test-slave-706q"
server_key: "2352272790dadc90b756be6f6331a30b"
server_key_iv: "f207046b95b808e08c42457a1be6f33f"
server_key_version: "encryptionkey@0"
I20260620 09:30:18.220450 5077 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 09:30:18.221277 5092 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:30:18.221477 5077 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 09:30:18.221583 5077 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781947562949448-16558-0/minicluster-data/master-0/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781947562949448-16558-0/minicluster-data/master-0/wal
uuid: "b7295a7448a148aa9364938832f805f9"
format_stamp: "Formatted at 2026-06-20 09:30:18 on dist-test-slave-706q"
server_key: "2352272790dadc90b756be6f6331a30b"
server_key_iv: "f207046b95b808e08c42457a1be6f33f"
server_key_version: "encryptionkey@0"
I20260620 09:30:18.221662 5077 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781947562949448-16558-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781947562949448-16558-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781947562949448-16558-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:30:18.259682 5077 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:30:18.261983 5077 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:30:18.262182 5077 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:30:18.266492 5077 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36179
I20260620 09:30:18.266549 5144 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36179 every 8 connection(s)
I20260620 09:30:18.267081 5077 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
I20260620 09:30:18.267612 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 5077
I20260620 09:30:18.267743 16558 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781947562949448-16558-0/minicluster-data/master-0/wal/instance
I20260620 09:30:18.267974 5145 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:30:18.268011 16558 external_mini_cluster.cc:1428] Setting key 09780d0dbaf0f6ba9d7c9445491b8921
I20260620 09:30:18.271023 5145 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7295a7448a148aa9364938832f805f9: Bootstrap starting.
I20260620 09:30:18.271803 5145 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b7295a7448a148aa9364938832f805f9: Neither blocks nor log segments found. Creating new log.
I20260620 09:30:18.272081 5145 log.cc:824] T 00000000000000000000000000000000 P b7295a7448a148aa9364938832f805f9: Log is configured to *not* fsync() on all Append() calls
I20260620 09:30:18.272733 5145 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7295a7448a148aa9364938832f805f9: No bootstrap required, opened a new log
I20260620 09:30:18.274047 5148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:18.268622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.41:51993 (local address 127.0.0.1:36179)
0620 09:30:18.268788 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:18.268803 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:18.268835 (+ 32us) server_negotiation.cc:407] Connection header received
0620 09:30:18.269402 (+ 567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:18.269409 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:18.269455 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:18.269571 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:30:18.269887 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:18.270759 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:18.271917 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:18.272401 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:18.272523 (+ 122us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:18.273340 (+ 817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:30:18.273344 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:30:18.273345 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:30:18.273520 (+ 175us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:30:18.273554 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:30:18.273565 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:30:18.273659 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:18.273754 (+ 95us) server_negotiation.cc:299] Negotiation successful
0620 09:30:18.273783 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":89,"threads_started":1}
I20260620 09:30:18.274763 5145 raft_consensus.cc:348] T 00000000000000000000000000000000 P b7295a7448a148aa9364938832f805f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7295a7448a148aa9364938832f805f9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36179 } }
I20260620 09:30:18.274920 5145 raft_consensus.cc:374] T 00000000000000000000000000000000 P b7295a7448a148aa9364938832f805f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:30:18.274974 5145 raft_consensus.cc:729] T 00000000000000000000000000000000 P b7295a7448a148aa9364938832f805f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7295a7448a148aa9364938832f805f9, State: Initialized, Role: FOLLOWER
I20260620 09:30:18.275031 16558 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 09:30:18.275120 5145 consensus_queue.cc:260] T 00000000000000000000000000000000 P b7295a7448a148aa9364938832f805f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7295a7448a148aa9364938832f805f9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36179 } }
I20260620 09:30:18.275230 5145 raft_consensus.cc:388] T 00000000000000000000000000000000 P b7295a7448a148aa9364938832f805f9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:30:18.275274 5145 raft_consensus.cc:482] T 00000000000000000000000000000000 P b7295a7448a148aa9364938832f805f9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:30:18.275322 5145 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b7295a7448a148aa9364938832f805f9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:30:18.275960 5145 raft_consensus.cc:504] T 00000000000000000000000000000000 P b7295a7448a148aa9364938832f805f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7295a7448a148aa9364938832f805f9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36179 } }
I20260620 09:30:18.276108 5145 leader_election.cc:302] T 00000000000000000000000000000000 P b7295a7448a148aa9364938832f805f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b7295a7448a148aa9364938832f805f9; no voters:
I20260620 09:30:18.276281 5145 leader_election.cc:288] T 00000000000000000000000000000000 P b7295a7448a148aa9364938832f805f9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:30:18.276314 5154 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b7295a7448a148aa9364938832f805f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:30:18.276487 5154 raft_consensus.cc:686] T 00000000000000000000000000000000 P b7295a7448a148aa9364938832f805f9 [term 1 LEADER]: Becoming Leader. State: Replica: b7295a7448a148aa9364938832f805f9, State: Running, Role: LEADER
I20260620 09:30:18.276614 5145 sys_catalog.cc:564] T 00000000000000000000000000000000 P b7295a7448a148aa9364938832f805f9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:30:18.276620 5154 consensus_queue.cc:237] T 00000000000000000000000000000000 P b7295a7448a148aa9364938832f805f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7295a7448a148aa9364938832f805f9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36179 } }
I20260620 09:30:18.277091 5155 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7295a7448a148aa9364938832f805f9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b7295a7448a148aa9364938832f805f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7295a7448a148aa9364938832f805f9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36179 } } }
I20260620 09:30:18.277115 5156 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7295a7448a148aa9364938832f805f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b7295a7448a148aa9364938832f805f9. Latest consensus state: current_term: 1 leader_uuid: "b7295a7448a148aa9364938832f805f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7295a7448a148aa9364938832f805f9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36179 } } }
I20260620 09:30:18.277263 5156 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7295a7448a148aa9364938832f805f9 [sys.catalog]: This master's current role is: LEADER
I20260620 09:30:18.277252 5155 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7295a7448a148aa9364938832f805f9 [sys.catalog]: This master's current role is: LEADER
I20260620 09:30:18.277590 5164 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:30:18.278105 5164 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:30:18.280323 5148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:18.276509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.41:35469 (local address 127.0.0.1:36179)
0620 09:30:18.276548 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:18.276551 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:18.276724 (+ 173us) server_negotiation.cc:407] Connection header received
0620 09:30:18.276812 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:18.276813 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:18.276822 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:18.276849 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:30:18.277033 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:18.277763 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:18.278847 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:18.279136 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:18.279198 (+ 62us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:18.280058 (+ 860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:30:18.280060 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:30:18.280060 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:30:18.280065 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:30:18.280081 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:30:18.280086 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:30:18.280163 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:18.280227 (+ 64us) server_negotiation.cc:299] Negotiation successful
0620 09:30:18.280238 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 09:30:18.284013 5164 catalog_manager.cc:1347] Generated new cluster ID: e58bcd227d85427bb84299ca9884f62e
I20260620 09:30:18.284097 5164 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:30:18.295858 5164 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:30:18.296417 5164 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:30:18.300729 5164 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b7295a7448a148aa9364938832f805f9: Generated new TSK 0
I20260620 09:30:18.300962 5164 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 09:30:18.305447 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 5077
W20260620 09:30:18.309775 5062 connection.cc:535] client connection to 127.0.0.1:36179 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36179): Connection reset by peer (error 104)
2026-06-20T09:30:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1357 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781947562949448-16558-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:30:18 dist-test-slave-706q krb5kdc[5176](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:30:18 dist-test-slave-706q krb5kdc[5176](info): set up 2 sockets
Jun 20 09:30:18 dist-test-slave-706q krb5kdc[5176](info): commencing operation
krb5kdc: starting...
W20260620 09:30:19.357012 16558 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781947562949448-16558-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781947562949448-16558-0/krb5kdc/test-user.keytab.
Jun 20 09:30:19 dist-test-slave-706q krb5kdc[5176](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947819, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:30:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:30:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781947562949448-16558-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781947562949448-16558-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 09:30:19.482203 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781947562949448-16558-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781947562949448-16558-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781947562949448-16558-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35165
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781947562949448-16558-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35165
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781947562949448-16558-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781947562949448-16558-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781947562949448-16558-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:30:19.562994 5189 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:30:19.563246 5189 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:30:19.563283 5189 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:30:19.564461 5189 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:30:19.564535 5189 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:30:19.564560 5189 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:30:19.564589 5189 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:30:19.564618 5189 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:30:19.566118 5189 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781947562949448-16558-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781947562949448-16558-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35165
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781947562949448-16558-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35165
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5189
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781947562949448-16558-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:30:19.566398 5189 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:30:19.566623 5189 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:30:19.571728 5197 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:30:19.571760 5195 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:30:19.571772 5189 server_base.cc:1034] running on GCE node
W20260620 09:30:19.571739 5194 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:30:19.572283 5189 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:30:19.572589 5189 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:30:19.573773 5189 hybrid_clock.cc:648] HybridClock initialized: now 1781947819573750 us; error 40 us; skew 500 ppm
Jun 20 09:30:19 dist-test-slave-706q krb5kdc[5176](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947819, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:30:19.576471 5189 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 09:30:19.577013 5189 webserver.cc:460] Webserver started at http://127.0.0.1:36889/ using document root <none> and password file <none>
I20260620 09:30:19.577303 5189 fs_manager.cc:359] Metadata directory not provided
I20260620 09:30:19.577380 5189 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:30:19.577534 5189 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:30:19.578632 5189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781947562949448-16558-0/minicluster-data/master-0/data/instance:
uuid: "99453d83cc904cfda9b2b33de51c9999"
format_stamp: "Formatted at 2026-06-20 09:30:19 on dist-test-slave-706q"
server_key: "017ae3fffe5d08fd8ff1a58a17fbac5b"
server_key_iv: "4326978280adb16527ffd03cb4d2eeea"
server_key_version: "encryptionkey@0"
I20260620 09:30:19.579010 5189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781947562949448-16558-0/minicluster-data/master-0/wal/instance:
uuid: "99453d83cc904cfda9b2b33de51c9999"
format_stamp: "Formatted at 2026-06-20 09:30:19 on dist-test-slave-706q"
server_key: "017ae3fffe5d08fd8ff1a58a17fbac5b"
server_key_iv: "4326978280adb16527ffd03cb4d2eeea"
server_key_version: "encryptionkey@0"
I20260620 09:30:19.580493 5189 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 09:30:19.581350 5204 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:30:19.581583 5189 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 09:30:19.581704 5189 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781947562949448-16558-0/minicluster-data/master-0/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781947562949448-16558-0/minicluster-data/master-0/wal
uuid: "99453d83cc904cfda9b2b33de51c9999"
format_stamp: "Formatted at 2026-06-20 09:30:19 on dist-test-slave-706q"
server_key: "017ae3fffe5d08fd8ff1a58a17fbac5b"
server_key_iv: "4326978280adb16527ffd03cb4d2eeea"
server_key_version: "encryptionkey@0"
I20260620 09:30:19.581856 5189 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781947562949448-16558-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781947562949448-16558-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781947562949448-16558-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:30:19.585357 5189 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:30:19.587488 5189 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:30:19.587870 5189 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:30:19.591898 5189 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35165
I20260620 09:30:19.591928 5256 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35165 every 8 connection(s)
I20260620 09:30:19.592283 5189 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
I20260620 09:30:19.593171 5257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:30:19.595441 5257 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99453d83cc904cfda9b2b33de51c9999: Bootstrap starting.
I20260620 09:30:19.596079 5257 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 99453d83cc904cfda9b2b33de51c9999: Neither blocks nor log segments found. Creating new log.
I20260620 09:30:19.596374 5257 log.cc:824] T 00000000000000000000000000000000 P 99453d83cc904cfda9b2b33de51c9999: Log is configured to *not* fsync() on all Append() calls
I20260620 09:30:19.597213 5257 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99453d83cc904cfda9b2b33de51c9999: No bootstrap required, opened a new log
I20260620 09:30:19.597488 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 5189
I20260620 09:30:19.597621 16558 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781947562949448-16558-0/minicluster-data/master-0/wal/instance
I20260620 09:30:19.597877 16558 external_mini_cluster.cc:1428] Setting key 2b50c9d5d47722d7a5db8fa03dd18671
I20260620 09:30:19.599774 5257 raft_consensus.cc:348] T 00000000000000000000000000000000 P 99453d83cc904cfda9b2b33de51c9999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99453d83cc904cfda9b2b33de51c9999" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35165 } }
I20260620 09:30:19.599999 5257 raft_consensus.cc:374] T 00000000000000000000000000000000 P 99453d83cc904cfda9b2b33de51c9999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:30:19.600035 5257 raft_consensus.cc:729] T 00000000000000000000000000000000 P 99453d83cc904cfda9b2b33de51c9999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99453d83cc904cfda9b2b33de51c9999, State: Initialized, Role: FOLLOWER
I20260620 09:30:19.600162 5257 consensus_queue.cc:260] T 00000000000000000000000000000000 P 99453d83cc904cfda9b2b33de51c9999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99453d83cc904cfda9b2b33de51c9999" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35165 } }
I20260620 09:30:19.600255 5257 raft_consensus.cc:388] T 00000000000000000000000000000000 P 99453d83cc904cfda9b2b33de51c9999 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:30:19.600293 5257 raft_consensus.cc:482] T 00000000000000000000000000000000 P 99453d83cc904cfda9b2b33de51c9999 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:30:19.600389 5257 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 99453d83cc904cfda9b2b33de51c9999 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:30:19.601024 5257 raft_consensus.cc:504] T 00000000000000000000000000000000 P 99453d83cc904cfda9b2b33de51c9999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99453d83cc904cfda9b2b33de51c9999" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35165 } }
I20260620 09:30:19.601137 5257 leader_election.cc:302] T 00000000000000000000000000000000 P 99453d83cc904cfda9b2b33de51c9999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 99453d83cc904cfda9b2b33de51c9999; no voters:
I20260620 09:30:19.601334 5257 leader_election.cc:288] T 00000000000000000000000000000000 P 99453d83cc904cfda9b2b33de51c9999 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:30:19.601362 5262 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 99453d83cc904cfda9b2b33de51c9999 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:30:19.601537 5262 raft_consensus.cc:686] T 00000000000000000000000000000000 P 99453d83cc904cfda9b2b33de51c9999 [term 1 LEADER]: Becoming Leader. State: Replica: 99453d83cc904cfda9b2b33de51c9999, State: Running, Role: LEADER
I20260620 09:30:19.601671 5261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:19.598455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.41:46423 (local address 127.0.0.1:35165)
0620 09:30:19.598646 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:19.598659 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:19.598694 (+ 35us) server_negotiation.cc:407] Connection header received
0620 09:30:19.599978 (+ 1284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:19.599986 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:19.600024 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:19.600182 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:30:19.600996 (+ 814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:30:19.601000 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:30:19.601000 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:30:19.601005 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:30:19.601055 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:30:19.601069 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:30:19.601148 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:19.601397 (+ 249us) server_negotiation.cc:299] Negotiation successful
0620 09:30:19.601447 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":85,"threads_started":1}
I20260620 09:30:19.601679 5257 sys_catalog.cc:564] T 00000000000000000000000000000000 P 99453d83cc904cfda9b2b33de51c9999 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:30:19.601702 5262 consensus_queue.cc:237] T 00000000000000000000000000000000 P 99453d83cc904cfda9b2b33de51c9999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99453d83cc904cfda9b2b33de51c9999" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35165 } }
I20260620 09:30:19.602473 5264 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99453d83cc904cfda9b2b33de51c9999 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 99453d83cc904cfda9b2b33de51c9999. Latest consensus state: current_term: 1 leader_uuid: "99453d83cc904cfda9b2b33de51c9999" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99453d83cc904cfda9b2b33de51c9999" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35165 } } }
I20260620 09:30:19.602622 5264 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99453d83cc904cfda9b2b33de51c9999 [sys.catalog]: This master's current role is: LEADER
I20260620 09:30:19.602847 16558 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 09:30:19.603083 5271 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:30:19.603714 5271 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:30:19.604542 5263 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99453d83cc904cfda9b2b33de51c9999 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "99453d83cc904cfda9b2b33de51c9999" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99453d83cc904cfda9b2b33de51c9999" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35165 } } }
I20260620 09:30:19.604633 5263 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99453d83cc904cfda9b2b33de51c9999 [sys.catalog]: This master's current role is: LEADER
I20260620 09:30:19.605449 5271 catalog_manager.cc:1347] Generated new cluster ID: 7d57d50aeaef4d9fa9bcd726e76472a3
I20260620 09:30:19.605504 5271 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:30:19.613047 5261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:19.607956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.41:38519 (local address 127.0.0.1:35165)
0620 09:30:19.608000 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:19.608002 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:19.611844 (+ 3842us) server_negotiation.cc:407] Connection header received
0620 09:30:19.611958 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:19.611959 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:19.611965 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:19.612048 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:30:19.612782 (+ 734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:30:19.612784 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:30:19.612785 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:30:19.612789 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:30:19.612805 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:30:19.612809 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:30:19.612865 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:19.612922 (+ 57us) server_negotiation.cc:299] Negotiation successful
0620 09:30:19.612938 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 09:30:19.616796 5271 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:30:19.617115 5271 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:30:19.628508 5271 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 99453d83cc904cfda9b2b33de51c9999: Generated new TSK 0
I20260620 09:30:19.628731 5271 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 09:30:19.632983 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 5189
2026-06-20T09:30:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1328 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781947562949448-16558-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:30:19 dist-test-slave-706q krb5kdc[5288](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:30:19 dist-test-slave-706q krb5kdc[5288](info): set up 2 sockets
Jun 20 09:30:19 dist-test-slave-706q krb5kdc[5288](info): commencing operation
krb5kdc: starting...
W20260620 09:30:20.704756 16558 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781947562949448-16558-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781947562949448-16558-0/krb5kdc/test-user.keytab.
Jun 20 09:30:20 dist-test-slave-706q krb5kdc[5288](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947820, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:30:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:30:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781947562949448-16558-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781947562949448-16558-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 09:30:20.828358 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781947562949448-16558-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781947562949448-16558-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781947562949448-16558-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35183
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781947562949448-16558-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35183
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781947562949448-16558-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781947562949448-16558-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781947562949448-16558-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:30:20.912555 5301 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:30:20.912776 5301 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:30:20.912801 5301 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:30:20.913987 5301 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:30:20.914038 5301 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:30:20.914052 5301 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:30:20.914065 5301 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:30:20.914077 5301 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:30:20.915664 5301 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781947562949448-16558-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781947562949448-16558-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35183
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781947562949448-16558-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35183
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5301
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781947562949448-16558-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:30:20.915930 5301 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:30:20.916159 5301 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 09:30:20.921345 5301 server_base.cc:1034] running on GCE node
W20260620 09:30:20.921253 5309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:30:20.921249 5306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:30:20.921289 5307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:30:20.921829 5301 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:30:20.922137 5301 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:30:20.923308 5301 hybrid_clock.cc:648] HybridClock initialized: now 1781947820923286 us; error 60 us; skew 500 ppm
Jun 20 09:30:20 dist-test-slave-706q krb5kdc[5288](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947820, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:30:20.929468 5301 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 09:30:20.929984 5301 webserver.cc:460] Webserver started at http://127.0.0.1:34013/ using document root <none> and password file <none>
I20260620 09:30:20.930219 5301 fs_manager.cc:359] Metadata directory not provided
I20260620 09:30:20.930291 5301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:30:20.930454 5301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:30:20.931635 5301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781947562949448-16558-0/minicluster-data/master-0/data/instance:
uuid: "75535bd30add43d1b85d8dade1c205c1"
format_stamp: "Formatted at 2026-06-20 09:30:20 on dist-test-slave-706q"
server_key: "b0c80e9b1f20984cfca5e8813aaf2b79"
server_key_iv: "26e752e8c2b23b23edfc9347b9a1a267"
server_key_version: "encryptionkey@0"
I20260620 09:30:20.932003 5301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781947562949448-16558-0/minicluster-data/master-0/wal/instance:
uuid: "75535bd30add43d1b85d8dade1c205c1"
format_stamp: "Formatted at 2026-06-20 09:30:20 on dist-test-slave-706q"
server_key: "b0c80e9b1f20984cfca5e8813aaf2b79"
server_key_iv: "26e752e8c2b23b23edfc9347b9a1a267"
server_key_version: "encryptionkey@0"
I20260620 09:30:20.933542 5301 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 09:30:20.934502 5316 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:30:20.934751 5301 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 09:30:20.934880 5301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781947562949448-16558-0/minicluster-data/master-0/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781947562949448-16558-0/minicluster-data/master-0/wal
uuid: "75535bd30add43d1b85d8dade1c205c1"
format_stamp: "Formatted at 2026-06-20 09:30:20 on dist-test-slave-706q"
server_key: "b0c80e9b1f20984cfca5e8813aaf2b79"
server_key_iv: "26e752e8c2b23b23edfc9347b9a1a267"
server_key_version: "encryptionkey@0"
I20260620 09:30:20.934976 5301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781947562949448-16558-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781947562949448-16558-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781947562949448-16558-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:30:20.954169 5301 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:30:20.956540 5301 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:30:20.956719 5301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:30:20.961813 5301 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35183
I20260620 09:30:20.961855 5368 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35183 every 8 connection(s)
I20260620 09:30:20.962224 5301 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
I20260620 09:30:20.963165 5369 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:30:20.963364 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 5301
I20260620 09:30:20.963477 16558 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781947562949448-16558-0/minicluster-data/master-0/wal/instance
I20260620 09:30:20.963716 16558 external_mini_cluster.cc:1428] Setting key 9ae224b1350ab266d68fc2ab10850153
I20260620 09:30:20.966756 5369 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75535bd30add43d1b85d8dade1c205c1: Bootstrap starting.
I20260620 09:30:20.967500 5369 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 75535bd30add43d1b85d8dade1c205c1: Neither blocks nor log segments found. Creating new log.
I20260620 09:30:20.967846 5369 log.cc:824] T 00000000000000000000000000000000 P 75535bd30add43d1b85d8dade1c205c1: Log is configured to *not* fsync() on all Append() calls
I20260620 09:30:20.968606 5369 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75535bd30add43d1b85d8dade1c205c1: No bootstrap required, opened a new log
I20260620 09:30:20.970522 5369 raft_consensus.cc:348] T 00000000000000000000000000000000 P 75535bd30add43d1b85d8dade1c205c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75535bd30add43d1b85d8dade1c205c1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35183 } }
I20260620 09:30:20.970706 5369 raft_consensus.cc:374] T 00000000000000000000000000000000 P 75535bd30add43d1b85d8dade1c205c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:30:20.970741 5369 raft_consensus.cc:729] T 00000000000000000000000000000000 P 75535bd30add43d1b85d8dade1c205c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75535bd30add43d1b85d8dade1c205c1, State: Initialized, Role: FOLLOWER
Jun 20 09:30:20 dist-test-slave-706q krb5kdc[5288](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947820, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 09:30:20.970875 5369 consensus_queue.cc:260] T 00000000000000000000000000000000 P 75535bd30add43d1b85d8dade1c205c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75535bd30add43d1b85d8dade1c205c1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35183 } }
I20260620 09:30:20.970968 5369 raft_consensus.cc:388] T 00000000000000000000000000000000 P 75535bd30add43d1b85d8dade1c205c1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:30:20.971037 5369 raft_consensus.cc:482] T 00000000000000000000000000000000 P 75535bd30add43d1b85d8dade1c205c1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:30:20.971087 5369 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 75535bd30add43d1b85d8dade1c205c1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:30:20.971722 5369 raft_consensus.cc:504] T 00000000000000000000000000000000 P 75535bd30add43d1b85d8dade1c205c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75535bd30add43d1b85d8dade1c205c1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35183 } }
I20260620 09:30:20.971861 5369 leader_election.cc:302] T 00000000000000000000000000000000 P 75535bd30add43d1b85d8dade1c205c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 75535bd30add43d1b85d8dade1c205c1; no voters:
I20260620 09:30:20.972040 5369 leader_election.cc:288] T 00000000000000000000000000000000 P 75535bd30add43d1b85d8dade1c205c1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:30:20.972118 5374 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 75535bd30add43d1b85d8dade1c205c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:30:20.972343 5374 raft_consensus.cc:686] T 00000000000000000000000000000000 P 75535bd30add43d1b85d8dade1c205c1 [term 1 LEADER]: Becoming Leader. State: Replica: 75535bd30add43d1b85d8dade1c205c1, State: Running, Role: LEADER
I20260620 09:30:20.972368 5369 sys_catalog.cc:564] T 00000000000000000000000000000000 P 75535bd30add43d1b85d8dade1c205c1 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:30:20.972499 5374 consensus_queue.cc:237] T 00000000000000000000000000000000 P 75535bd30add43d1b85d8dade1c205c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75535bd30add43d1b85d8dade1c205c1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35183 } }
I20260620 09:30:20.972927 5375 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75535bd30add43d1b85d8dade1c205c1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "75535bd30add43d1b85d8dade1c205c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75535bd30add43d1b85d8dade1c205c1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35183 } } }
I20260620 09:30:20.972944 5376 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75535bd30add43d1b85d8dade1c205c1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 75535bd30add43d1b85d8dade1c205c1. Latest consensus state: current_term: 1 leader_uuid: "75535bd30add43d1b85d8dade1c205c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75535bd30add43d1b85d8dade1c205c1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35183 } } }
I20260620 09:30:20.973031 5375 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75535bd30add43d1b85d8dade1c205c1 [sys.catalog]: This master's current role is: LEADER
I20260620 09:30:20.973050 5376 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75535bd30add43d1b85d8dade1c205c1 [sys.catalog]: This master's current role is: LEADER
I20260620 09:30:20.973249 5381 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:30:20.973830 5381 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:30:20.975494 5381 catalog_manager.cc:1347] Generated new cluster ID: f73bf629b6684db19b4a843881259b57
I20260620 09:30:20.975574 5381 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:30:20.975894 5372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:20.964264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.41:40079 (local address 127.0.0.1:35183)
0620 09:30:20.964447 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:20.964460 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:20.964485 (+ 25us) server_negotiation.cc:407] Connection header received
0620 09:30:20.964974 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:20.964980 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:20.965039 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:20.965157 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:30:20.965992 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:20.966972 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:20.968115 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:20.968671 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:20.968886 (+ 215us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:20.971471 (+ 2585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:30:20.971477 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:30:20.971478 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:30:20.971494 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:30:20.973901 (+ 2407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:30:20.974562 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:30:20.974565 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:30:20.974566 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:30:20.974724 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:30:20.974980 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:30:20.974981 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:30:20.974981 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:30:20.975272 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:30:20.975459 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:20.975626 (+ 167us) server_negotiation.cc:299] Negotiation successful
0620 09:30:20.975729 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":99,"threads_started":1}
I20260620 09:30:20.976743 16558 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 09:30:20.986049 5372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:20.979662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.41:39183 (local address 127.0.0.1:35183)
0620 09:30:20.979727 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:20.979729 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:20.979739 (+ 10us) server_negotiation.cc:407] Connection header received
0620 09:30:20.979763 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:20.979764 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:20.979784 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:20.979860 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:30:20.980435 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:20.980942 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:20.981640 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:20.981881 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:20.981951 (+ 70us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:20.983156 (+ 1205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:30:20.983158 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:30:20.983158 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:30:20.983165 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:30:20.984862 (+ 1697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:30:20.985249 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:30:20.985251 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:30:20.985251 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:30:20.985309 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:30:20.985537 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:30:20.985539 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:30:20.985539 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:30:20.985649 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:30:20.985759 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:20.985880 (+ 121us) server_negotiation.cc:299] Negotiation successful
0620 09:30:20.985943 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 09:30:20.992923 5381 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:30:20.993650 5381 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:30:20.998260 5381 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 75535bd30add43d1b85d8dade1c205c1: Generated new TSK 0
I20260620 09:30:20.998471 5381 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 09:30:21.005846 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 5301
W20260620 09:30:21.010275 5286 connection.cc:535] client connection to 127.0.0.1:35183 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35183): Connection reset by peer (error 104)
2026-06-20T09:30:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1372 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781947562949448-16558-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:30:21 dist-test-slave-706q krb5kdc[5400](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:30:21 dist-test-slave-706q krb5kdc[5400](info): set up 2 sockets
Jun 20 09:30:21 dist-test-slave-706q krb5kdc[5400](info): commencing operation
krb5kdc: starting...
W20260620 09:30:22.040786 16558 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781947562949448-16558-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781947562949448-16558-0/krb5kdc/test-user.keytab.
Jun 20 09:30:22 dist-test-slave-706q krb5kdc[5400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947822, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:30:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:30:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781947562949448-16558-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781947562949448-16558-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 09:30:22.167188 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781947562949448-16558-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781947562949448-16558-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781947562949448-16558-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37043
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781947562949448-16558-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37043
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781947562949448-16558-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781947562949448-16558-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781947562949448-16558-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:30:22.248046 5413 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:30:22.248325 5413 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:30:22.248365 5413 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:30:22.249727 5413 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:30:22.249809 5413 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:30:22.249833 5413 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:30:22.249856 5413 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:30:22.249886 5413 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:30:22.251415 5413 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781947562949448-16558-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781947562949448-16558-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37043
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781947562949448-16558-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37043
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5413
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781947562949448-16558-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:30:22.251736 5413 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:30:22.252002 5413 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:30:22.257133 5421 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:30:22.257172 5418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:30:22.257201 5419 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:30:22.257405 5413 server_base.cc:1034] running on GCE node
I20260620 09:30:22.257712 5413 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:30:22.258034 5413 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:30:22.259178 5413 hybrid_clock.cc:648] HybridClock initialized: now 1781947822259143 us; error 62 us; skew 500 ppm
Jun 20 09:30:22 dist-test-slave-706q krb5kdc[5400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947822, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:30:22.261754 5413 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 09:30:22.262248 5413 webserver.cc:460] Webserver started at http://127.0.0.1:34955/ using document root <none> and password file <none>
I20260620 09:30:22.262483 5413 fs_manager.cc:359] Metadata directory not provided
I20260620 09:30:22.262552 5413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:30:22.262709 5413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:30:22.263870 5413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781947562949448-16558-0/minicluster-data/master-0/data/instance:
uuid: "eb14fdefe4d44cf2ba013573f395cc48"
format_stamp: "Formatted at 2026-06-20 09:30:22 on dist-test-slave-706q"
server_key: "6a01de7140623381cfa9e38cedeb8487"
server_key_iv: "83dcea679a8ce8ac63e0c7e43cdc4e95"
server_key_version: "encryptionkey@0"
I20260620 09:30:22.264261 5413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781947562949448-16558-0/minicluster-data/master-0/wal/instance:
uuid: "eb14fdefe4d44cf2ba013573f395cc48"
format_stamp: "Formatted at 2026-06-20 09:30:22 on dist-test-slave-706q"
server_key: "6a01de7140623381cfa9e38cedeb8487"
server_key_iv: "83dcea679a8ce8ac63e0c7e43cdc4e95"
server_key_version: "encryptionkey@0"
I20260620 09:30:22.265678 5413 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260620 09:30:22.266490 5428 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:30:22.266710 5413 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:30:22.266819 5413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781947562949448-16558-0/minicluster-data/master-0/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781947562949448-16558-0/minicluster-data/master-0/wal
uuid: "eb14fdefe4d44cf2ba013573f395cc48"
format_stamp: "Formatted at 2026-06-20 09:30:22 on dist-test-slave-706q"
server_key: "6a01de7140623381cfa9e38cedeb8487"
server_key_iv: "83dcea679a8ce8ac63e0c7e43cdc4e95"
server_key_version: "encryptionkey@0"
I20260620 09:30:22.266892 5413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781947562949448-16558-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781947562949448-16558-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781947562949448-16558-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:30:22.278375 5413 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:30:22.280807 5413 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:30:22.280994 5413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:30:22.285378 5413 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37043
I20260620 09:30:22.285415 5480 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37043 every 8 connection(s)
I20260620 09:30:22.285785 5413 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
I20260620 09:30:22.286687 5481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:30:22.289563 5481 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb14fdefe4d44cf2ba013573f395cc48: Bootstrap starting.
I20260620 09:30:22.290268 5481 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eb14fdefe4d44cf2ba013573f395cc48: Neither blocks nor log segments found. Creating new log.
I20260620 09:30:22.290584 5481 log.cc:824] T 00000000000000000000000000000000 P eb14fdefe4d44cf2ba013573f395cc48: Log is configured to *not* fsync() on all Append() calls
I20260620 09:30:22.291438 5481 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb14fdefe4d44cf2ba013573f395cc48: No bootstrap required, opened a new log
I20260620 09:30:22.292317 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 5413
I20260620 09:30:22.292447 16558 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781947562949448-16558-0/minicluster-data/master-0/wal/instance
I20260620 09:30:22.292642 16558 external_mini_cluster.cc:1428] Setting key 402bf45b6a4819abe583c9a6c7c1aead
I20260620 09:30:22.293808 5481 raft_consensus.cc:348] T 00000000000000000000000000000000 P eb14fdefe4d44cf2ba013573f395cc48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb14fdefe4d44cf2ba013573f395cc48" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37043 } }
I20260620 09:30:22.293994 5481 raft_consensus.cc:374] T 00000000000000000000000000000000 P eb14fdefe4d44cf2ba013573f395cc48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:30:22.294034 5481 raft_consensus.cc:729] T 00000000000000000000000000000000 P eb14fdefe4d44cf2ba013573f395cc48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb14fdefe4d44cf2ba013573f395cc48, State: Initialized, Role: FOLLOWER
I20260620 09:30:22.294222 5481 consensus_queue.cc:260] T 00000000000000000000000000000000 P eb14fdefe4d44cf2ba013573f395cc48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb14fdefe4d44cf2ba013573f395cc48" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37043 } }
I20260620 09:30:22.294322 5481 raft_consensus.cc:388] T 00000000000000000000000000000000 P eb14fdefe4d44cf2ba013573f395cc48 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:30:22.294348 5481 raft_consensus.cc:482] T 00000000000000000000000000000000 P eb14fdefe4d44cf2ba013573f395cc48 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:30:22.294421 5481 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eb14fdefe4d44cf2ba013573f395cc48 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:30:22.295060 5481 raft_consensus.cc:504] T 00000000000000000000000000000000 P eb14fdefe4d44cf2ba013573f395cc48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb14fdefe4d44cf2ba013573f395cc48" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37043 } }
I20260620 09:30:22.295150 5481 leader_election.cc:302] T 00000000000000000000000000000000 P eb14fdefe4d44cf2ba013573f395cc48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eb14fdefe4d44cf2ba013573f395cc48; no voters:
I20260620 09:30:22.295359 5481 leader_election.cc:288] T 00000000000000000000000000000000 P eb14fdefe4d44cf2ba013573f395cc48 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:30:22.295382 5486 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eb14fdefe4d44cf2ba013573f395cc48 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:30:22.295499 5486 raft_consensus.cc:686] T 00000000000000000000000000000000 P eb14fdefe4d44cf2ba013573f395cc48 [term 1 LEADER]: Becoming Leader. State: Replica: eb14fdefe4d44cf2ba013573f395cc48, State: Running, Role: LEADER
I20260620 09:30:22.295653 5481 sys_catalog.cc:564] T 00000000000000000000000000000000 P eb14fdefe4d44cf2ba013573f395cc48 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:30:22.295671 5486 consensus_queue.cc:237] T 00000000000000000000000000000000 P eb14fdefe4d44cf2ba013573f395cc48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb14fdefe4d44cf2ba013573f395cc48" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37043 } }
I20260620 09:30:22.297412 5488 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb14fdefe4d44cf2ba013573f395cc48 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eb14fdefe4d44cf2ba013573f395cc48. Latest consensus state: current_term: 1 leader_uuid: "eb14fdefe4d44cf2ba013573f395cc48" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb14fdefe4d44cf2ba013573f395cc48" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37043 } } }
I20260620 09:30:22.297456 5487 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb14fdefe4d44cf2ba013573f395cc48 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eb14fdefe4d44cf2ba013573f395cc48" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb14fdefe4d44cf2ba013573f395cc48" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37043 } } }
I20260620 09:30:22.297521 5488 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb14fdefe4d44cf2ba013573f395cc48 [sys.catalog]: This master's current role is: LEADER
I20260620 09:30:22.297530 5487 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb14fdefe4d44cf2ba013573f395cc48 [sys.catalog]: This master's current role is: LEADER
W20260620 09:30:22.298272 5500 catalog_manager.cc:1558] T 00000000000000000000000000000000 P eb14fdefe4d44cf2ba013573f395cc48: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 09:30:22.298333 5500 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 09:30:22.298452 5501 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:30:22.298650 5501 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:30:22.299360 5485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:22.293223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.41:42045 (local address 127.0.0.1:37043)
0620 09:30:22.293422 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:22.293430 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:22.293458 (+ 28us) server_negotiation.cc:407] Connection header received
0620 09:30:22.293785 (+ 327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:22.293791 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:22.293851 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:22.293980 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:30:22.294284 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:22.295762 (+ 1478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:22.296932 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:22.297507 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:22.297630 (+ 123us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:22.298642 (+ 1012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:30:22.298646 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:30:22.298646 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:30:22.298875 (+ 229us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:30:22.298915 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:30:22.298930 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:30:22.299021 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:22.299108 (+ 87us) server_negotiation.cc:299] Negotiation successful
0620 09:30:22.299137 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":96,"threads_started":1}
I20260620 09:30:22.300398 5501 catalog_manager.cc:1347] Generated new cluster ID: 9dca1efb86f748b1b5fdaf60dac9d110
I20260620 09:30:22.300449 5501 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:30:22.300468 16558 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 09:30:22.310832 5501 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:30:22.310832 5485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:22.301989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.41:38301 (local address 127.0.0.1:37043)
0620 09:30:22.302024 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:22.302026 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:22.302122 (+ 96us) server_negotiation.cc:407] Connection header received
0620 09:30:22.302170 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:22.302171 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:22.302181 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:22.302238 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:30:22.302409 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:22.303180 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:22.304212 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:22.304609 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:22.304703 (+ 94us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:22.310387 (+ 5684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:30:22.310391 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:30:22.310391 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:30:22.310420 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:30:22.310440 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:30:22.310447 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:30:22.310522 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:22.310713 (+ 191us) server_negotiation.cc:299] Negotiation successful
0620 09:30:22.310729 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":12,"server-negotiator.queue_time_us":18}
I20260620 09:30:22.311640 5501 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:30:22.316494 5501 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eb14fdefe4d44cf2ba013573f395cc48: Generated new TSK 0
I20260620 09:30:22.316704 5501 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 09:30:22.319339 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 5413
W20260620 09:30:22.323871 5398 connection.cc:535] client connection to 127.0.0.1:37043 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37043): Connection reset by peer (error 104)
2026-06-20T09:30:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1313 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781947562949448-16558-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:30:22 dist-test-slave-706q krb5kdc[5512](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:30:22 dist-test-slave-706q krb5kdc[5512](info): set up 2 sockets
Jun 20 09:30:22 dist-test-slave-706q krb5kdc[5512](info): commencing operation
krb5kdc: starting...
W20260620 09:30:23.372810 16558 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781947562949448-16558-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781947562949448-16558-0/krb5kdc/test-user.keytab.
Jun 20 09:30:23 dist-test-slave-706q krb5kdc[5512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947823, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:30:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:30:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781947562949448-16558-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781947562949448-16558-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 09:30:23.498939 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781947562949448-16558-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781947562949448-16558-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781947562949448-16558-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37873
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781947562949448-16558-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37873
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781947562949448-16558-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781947562949448-16558-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781947562949448-16558-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:30:23.580976 5525 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:30:23.581248 5525 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:30:23.581290 5525 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:30:23.582505 5525 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:30:23.582583 5525 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:30:23.582607 5525 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:30:23.582628 5525 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:30:23.582659 5525 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:30:23.584246 5525 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781947562949448-16558-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781947562949448-16558-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37873
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781947562949448-16558-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37873
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5525
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781947562949448-16558-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:30:23.584550 5525 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:30:23.584798 5525 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:30:23.589911 5533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:30:23.589895 5531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:30:23.589900 5530 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:30:23.590342 5525 server_base.cc:1034] running on GCE node
I20260620 09:30:23.590641 5525 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:30:23.590998 5525 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:30:23.592168 5525 hybrid_clock.cc:648] HybridClock initialized: now 1781947823592160 us; error 54 us; skew 500 ppm
Jun 20 09:30:23 dist-test-slave-706q krb5kdc[5512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947823, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:30:23.594817 5525 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 09:30:23.595335 5525 webserver.cc:460] Webserver started at http://127.0.0.1:36057/ using document root <none> and password file <none>
I20260620 09:30:23.595631 5525 fs_manager.cc:359] Metadata directory not provided
I20260620 09:30:23.595702 5525 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:30:23.595867 5525 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:30:23.596974 5525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781947562949448-16558-0/minicluster-data/master-0/data/instance:
uuid: "20ab7628d30a42da9564d5067306c123"
format_stamp: "Formatted at 2026-06-20 09:30:23 on dist-test-slave-706q"
server_key: "32aa759ebd42b7a5782b2b171ebf2ab4"
server_key_iv: "2e5140d02b8a198c63b670ea4da1f458"
server_key_version: "encryptionkey@0"
I20260620 09:30:23.597362 5525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781947562949448-16558-0/minicluster-data/master-0/wal/instance:
uuid: "20ab7628d30a42da9564d5067306c123"
format_stamp: "Formatted at 2026-06-20 09:30:23 on dist-test-slave-706q"
server_key: "32aa759ebd42b7a5782b2b171ebf2ab4"
server_key_iv: "2e5140d02b8a198c63b670ea4da1f458"
server_key_version: "encryptionkey@0"
I20260620 09:30:23.598886 5525 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 09:30:23.599769 5540 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:30:23.599960 5525 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:30:23.600068 5525 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781947562949448-16558-0/minicluster-data/master-0/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781947562949448-16558-0/minicluster-data/master-0/wal
uuid: "20ab7628d30a42da9564d5067306c123"
format_stamp: "Formatted at 2026-06-20 09:30:23 on dist-test-slave-706q"
server_key: "32aa759ebd42b7a5782b2b171ebf2ab4"
server_key_iv: "2e5140d02b8a198c63b670ea4da1f458"
server_key_version: "encryptionkey@0"
I20260620 09:30:23.600144 5525 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781947562949448-16558-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781947562949448-16558-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781947562949448-16558-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:30:23.603482 5525 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:30:23.605852 5525 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:30:23.606014 5525 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:30:23.610249 5525 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37873
I20260620 09:30:23.610285 5592 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37873 every 8 connection(s)
I20260620 09:30:23.610651 5525 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
I20260620 09:30:23.611651 5593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:30:23.613902 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 5525
I20260620 09:30:23.614029 16558 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781947562949448-16558-0/minicluster-data/master-0/wal/instance
I20260620 09:30:23.614243 16558 external_mini_cluster.cc:1428] Setting key 18805fb497689d8f5201013d3495009e
I20260620 09:30:23.614339 5593 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20ab7628d30a42da9564d5067306c123: Bootstrap starting.
I20260620 09:30:23.615065 5593 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 20ab7628d30a42da9564d5067306c123: Neither blocks nor log segments found. Creating new log.
I20260620 09:30:23.615401 5593 log.cc:824] T 00000000000000000000000000000000 P 20ab7628d30a42da9564d5067306c123: Log is configured to *not* fsync() on all Append() calls
I20260620 09:30:23.616199 5593 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20ab7628d30a42da9564d5067306c123: No bootstrap required, opened a new log
I20260620 09:30:23.616957 5596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:23.614787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.41:59933 (local address 127.0.0.1:37873)
0620 09:30:23.614985 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:23.614992 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:23.615016 (+ 24us) server_negotiation.cc:407] Connection header received
0620 09:30:23.615505 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:23.615514 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:23.615588 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:23.615760 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:30:23.616502 (+ 742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:30:23.616505 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:30:23.616505 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:30:23.616509 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:30:23.616544 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:30:23.616554 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:30:23.616620 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:23.616697 (+ 77us) server_negotiation.cc:299] Negotiation successful
0620 09:30:23.616750 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":102,"threads_started":1}
I20260620 09:30:23.617966 16558 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 09:30:23.618952 5593 raft_consensus.cc:348] T 00000000000000000000000000000000 P 20ab7628d30a42da9564d5067306c123 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20ab7628d30a42da9564d5067306c123" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37873 } }
I20260620 09:30:23.619153 5593 raft_consensus.cc:374] T 00000000000000000000000000000000 P 20ab7628d30a42da9564d5067306c123 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:30:23.619200 5593 raft_consensus.cc:729] T 00000000000000000000000000000000 P 20ab7628d30a42da9564d5067306c123 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20ab7628d30a42da9564d5067306c123, State: Initialized, Role: FOLLOWER
I20260620 09:30:23.619387 5593 consensus_queue.cc:260] T 00000000000000000000000000000000 P 20ab7628d30a42da9564d5067306c123 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20ab7628d30a42da9564d5067306c123" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37873 } }
I20260620 09:30:23.619496 5593 raft_consensus.cc:388] T 00000000000000000000000000000000 P 20ab7628d30a42da9564d5067306c123 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:30:23.619591 5593 raft_consensus.cc:482] T 00000000000000000000000000000000 P 20ab7628d30a42da9564d5067306c123 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:30:23.619647 5593 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 20ab7628d30a42da9564d5067306c123 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:30:23.620272 5593 raft_consensus.cc:504] T 00000000000000000000000000000000 P 20ab7628d30a42da9564d5067306c123 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20ab7628d30a42da9564d5067306c123" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37873 } }
I20260620 09:30:23.620365 5593 leader_election.cc:302] T 00000000000000000000000000000000 P 20ab7628d30a42da9564d5067306c123 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 20ab7628d30a42da9564d5067306c123; no voters:
I20260620 09:30:23.620551 5593 leader_election.cc:288] T 00000000000000000000000000000000 P 20ab7628d30a42da9564d5067306c123 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:30:23.620602 5603 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 20ab7628d30a42da9564d5067306c123 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:30:23.620811 5603 raft_consensus.cc:686] T 00000000000000000000000000000000 P 20ab7628d30a42da9564d5067306c123 [term 1 LEADER]: Becoming Leader. State: Replica: 20ab7628d30a42da9564d5067306c123, State: Running, Role: LEADER
I20260620 09:30:23.620887 5593 sys_catalog.cc:564] T 00000000000000000000000000000000 P 20ab7628d30a42da9564d5067306c123 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:30:23.621003 5603 consensus_queue.cc:237] T 00000000000000000000000000000000 P 20ab7628d30a42da9564d5067306c123 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20ab7628d30a42da9564d5067306c123" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37873 } }
I20260620 09:30:23.621250 5596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:23.619716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.41:47147 (local address 127.0.0.1:37873)
0620 09:30:23.619767 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:23.619769 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:23.619989 (+ 220us) server_negotiation.cc:407] Connection header received
0620 09:30:23.620084 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:23.620085 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:23.620091 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:23.620154 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:30:23.620974 (+ 820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:30:23.620975 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:30:23.620976 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:30:23.620982 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:30:23.620997 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:30:23.621001 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:30:23.621037 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:23.621157 (+ 120us) server_negotiation.cc:299] Negotiation successful
0620 09:30:23.621184 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 09:30:23.621507 5605 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20ab7628d30a42da9564d5067306c123 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "20ab7628d30a42da9564d5067306c123" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20ab7628d30a42da9564d5067306c123" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37873 } } }
I20260620 09:30:23.621539 5604 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20ab7628d30a42da9564d5067306c123 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 20ab7628d30a42da9564d5067306c123. Latest consensus state: current_term: 1 leader_uuid: "20ab7628d30a42da9564d5067306c123" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20ab7628d30a42da9564d5067306c123" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37873 } } }
I20260620 09:30:23.621660 5605 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20ab7628d30a42da9564d5067306c123 [sys.catalog]: This master's current role is: LEADER
I20260620 09:30:23.621765 5604 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20ab7628d30a42da9564d5067306c123 [sys.catalog]: This master's current role is: LEADER
I20260620 09:30:23.622057 5612 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:30:23.622599 5612 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:30:23.624396 5612 catalog_manager.cc:1347] Generated new cluster ID: a694eeab7d1f487cb29229e2ee53d17e
I20260620 09:30:23.624456 5612 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:30:23.641146 5612 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:30:23.641372 5612 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:30:23.647473 5612 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 20ab7628d30a42da9564d5067306c123: Generated new TSK 0
I20260620 09:30:23.647737 5612 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 09:30:23.652117 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 5525
2026-06-20T09:30:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1332 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781947562949448-16558-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:30:23 dist-test-slave-706q krb5kdc[5624](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:30:23 dist-test-slave-706q krb5kdc[5624](info): set up 2 sockets
Jun 20 09:30:23 dist-test-slave-706q krb5kdc[5624](info): commencing operation
krb5kdc: starting...
W20260620 09:30:24.700580 16558 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781947562949448-16558-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781947562949448-16558-0/krb5kdc/test-user.keytab.
Jun 20 09:30:24 dist-test-slave-706q krb5kdc[5624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947824, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:30:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:30:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.43.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.43.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.43.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.190.keytab.
Entry for principal kudu/127.16.43.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.190.keytab.
I20260620 09:30:24.822290 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781947562949448-16558-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781947562949448-16558-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781947562949448-16558-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.43.190:37495
--webserver_interface=127.16.43.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.190.keytab
--principal=kudu/127.16.43.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.43.148:45873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.43.190:37495
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781947562949448-16558-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781947562949448-16558-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781947562949448-16558-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:30:24.904223 5637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:30:24.904472 5637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:30:24.904515 5637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:30:24.905715 5637 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:30:24.905786 5637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:30:24.905812 5637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:30:24.905850 5637 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:30:24.905879 5637 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:30:24.907461 5637 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:45873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781947562949448-16558-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781947562949448-16558-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.43.190:37495
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.190.keytab
--principal=kudu/127.16.43.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.43.190:37495
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.43.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5637
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781947562949448-16558-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:30:24.907748 5637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:30:24.907989 5637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:30:24.913316 5643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:30:24.913326 5642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:30:24.913332 5645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:30:24.913508 5637 server_base.cc:1034] running on GCE node
I20260620 09:30:24.913887 5637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:30:24.914145 5637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:30:24.915334 5637 hybrid_clock.cc:648] HybridClock initialized: now 1781947824915303 us; error 48 us; skew 500 ppm
Jun 20 09:30:24 dist-test-slave-706q krb5kdc[5624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947824, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:30:24.918378 5637 init.cc:377] Logged in from keytab as kudu/127.16.43.190@KRBTEST.COM (short username kudu)
I20260620 09:30:24.918880 5637 webserver.cc:460] Webserver started at http://127.16.43.190:36177/ using document root <none> and password file <none>
I20260620 09:30:24.919134 5637 fs_manager.cc:359] Metadata directory not provided
I20260620 09:30:24.919209 5637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:30:24.919407 5637 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:30:24.920605 5637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781947562949448-16558-0/minicluster-data/master-0/data/instance:
uuid: "c2dc7f73b2a245789a258ce56fae12fe"
format_stamp: "Formatted at 2026-06-20 09:30:24 on dist-test-slave-706q"
server_key: "21e793b9e8abeedd18ee6287501ff07b"
server_key_iv: "36382e713baaccc6293ab9495435e866"
server_key_version: "encryptionkey@0"
I20260620 09:30:24.921002 5637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781947562949448-16558-0/minicluster-data/master-0/wal/instance:
uuid: "c2dc7f73b2a245789a258ce56fae12fe"
format_stamp: "Formatted at 2026-06-20 09:30:24 on dist-test-slave-706q"
server_key: "21e793b9e8abeedd18ee6287501ff07b"
server_key_iv: "36382e713baaccc6293ab9495435e866"
server_key_version: "encryptionkey@0"
I20260620 09:30:24.922456 5637 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 09:30:24.923393 5652 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:30:24.923638 5637 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 09:30:24.923802 5637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781947562949448-16558-0/minicluster-data/master-0/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781947562949448-16558-0/minicluster-data/master-0/wal
uuid: "c2dc7f73b2a245789a258ce56fae12fe"
format_stamp: "Formatted at 2026-06-20 09:30:24 on dist-test-slave-706q"
server_key: "21e793b9e8abeedd18ee6287501ff07b"
server_key_iv: "36382e713baaccc6293ab9495435e866"
server_key_version: "encryptionkey@0"
I20260620 09:30:24.923894 5637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781947562949448-16558-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781947562949448-16558-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781947562949448-16558-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:30:24.931200 5637 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:30:24.933525 5637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:30:24.933696 5637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:30:24.937899 5637 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.190:37495
I20260620 09:30:24.938050 5704 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.190:37495 every 8 connection(s)
I20260620 09:30:24.938323 5637 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
I20260620 09:30:24.939188 5705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:30:24.941540 5705 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2dc7f73b2a245789a258ce56fae12fe: Bootstrap starting.
I20260620 09:30:24.942119 5705 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c2dc7f73b2a245789a258ce56fae12fe: Neither blocks nor log segments found. Creating new log.
I20260620 09:30:24.942415 5705 log.cc:824] T 00000000000000000000000000000000 P c2dc7f73b2a245789a258ce56fae12fe: Log is configured to *not* fsync() on all Append() calls
I20260620 09:30:24.943161 5705 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2dc7f73b2a245789a258ce56fae12fe: No bootstrap required, opened a new log
I20260620 09:30:24.944836 5705 raft_consensus.cc:348] T 00000000000000000000000000000000 P c2dc7f73b2a245789a258ce56fae12fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2dc7f73b2a245789a258ce56fae12fe" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 37495 } }
I20260620 09:30:24.944979 5705 raft_consensus.cc:374] T 00000000000000000000000000000000 P c2dc7f73b2a245789a258ce56fae12fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:30:24.945003 5705 raft_consensus.cc:729] T 00000000000000000000000000000000 P c2dc7f73b2a245789a258ce56fae12fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2dc7f73b2a245789a258ce56fae12fe, State: Initialized, Role: FOLLOWER
I20260620 09:30:24.945159 5705 consensus_queue.cc:260] T 00000000000000000000000000000000 P c2dc7f73b2a245789a258ce56fae12fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2dc7f73b2a245789a258ce56fae12fe" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 37495 } }
I20260620 09:30:24.945225 5705 raft_consensus.cc:388] T 00000000000000000000000000000000 P c2dc7f73b2a245789a258ce56fae12fe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:30:24.945245 5705 raft_consensus.cc:482] T 00000000000000000000000000000000 P c2dc7f73b2a245789a258ce56fae12fe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:30:24.945317 5705 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c2dc7f73b2a245789a258ce56fae12fe [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:30:24.946183 5705 raft_consensus.cc:504] T 00000000000000000000000000000000 P c2dc7f73b2a245789a258ce56fae12fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2dc7f73b2a245789a258ce56fae12fe" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 37495 } }
I20260620 09:30:24.946309 5705 leader_election.cc:302] T 00000000000000000000000000000000 P c2dc7f73b2a245789a258ce56fae12fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c2dc7f73b2a245789a258ce56fae12fe; no voters:
I20260620 09:30:24.946491 5705 leader_election.cc:288] T 00000000000000000000000000000000 P c2dc7f73b2a245789a258ce56fae12fe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:30:24.946573 5708 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c2dc7f73b2a245789a258ce56fae12fe [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:30:24.946781 5708 raft_consensus.cc:686] T 00000000000000000000000000000000 P c2dc7f73b2a245789a258ce56fae12fe [term 1 LEADER]: Becoming Leader. State: Replica: c2dc7f73b2a245789a258ce56fae12fe, State: Running, Role: LEADER
I20260620 09:30:24.946789 5705 sys_catalog.cc:564] T 00000000000000000000000000000000 P c2dc7f73b2a245789a258ce56fae12fe [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:30:24.946883 5708 consensus_queue.cc:237] T 00000000000000000000000000000000 P c2dc7f73b2a245789a258ce56fae12fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2dc7f73b2a245789a258ce56fae12fe" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 37495 } }
I20260620 09:30:24.947299 5710 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2dc7f73b2a245789a258ce56fae12fe [sys.catalog]: SysCatalogTable state changed. Reason: New leader c2dc7f73b2a245789a258ce56fae12fe. Latest consensus state: current_term: 1 leader_uuid: "c2dc7f73b2a245789a258ce56fae12fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2dc7f73b2a245789a258ce56fae12fe" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 37495 } } }
I20260620 09:30:24.947407 5710 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2dc7f73b2a245789a258ce56fae12fe [sys.catalog]: This master's current role is: LEADER
I20260620 09:30:24.947788 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 5637
I20260620 09:30:24.947916 16558 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781947562949448-16558-0/minicluster-data/master-0/wal/instance
I20260620 09:30:24.948094 16558 external_mini_cluster.cc:1428] Setting key 0bcdb993c281c4f732c448ad7a35da51
W20260620 09:30:24.948582 5724 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c2dc7f73b2a245789a258ce56fae12fe: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 09:30:24.948634 5724 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 09:30:24.948694 5717 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:30:24.948743 5709 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2dc7f73b2a245789a258ce56fae12fe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c2dc7f73b2a245789a258ce56fae12fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2dc7f73b2a245789a258ce56fae12fe" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 37495 } } }
I20260620 09:30:24.948824 5709 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2dc7f73b2a245789a258ce56fae12fe [sys.catalog]: This master's current role is: LEADER
I20260620 09:30:24.948872 5717 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:30:24.950381 5717 catalog_manager.cc:1347] Generated new cluster ID: a8beeb6e6f984b05ac118db52a9131a6
I20260620 09:30:24.950435 5717 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 09:30:24 dist-test-slave-706q krb5kdc[5624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947824, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.43.190@KRBTEST.COM
I20260620 09:30:24.961675 5727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:24.948574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58261 (local address 127.16.43.190:37495)
0620 09:30:24.948943 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:24.948952 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:24.951859 (+ 2907us) server_negotiation.cc:407] Connection header received
0620 09:30:24.952039 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:24.952045 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:24.952090 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:24.952273 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:30:24.952937 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:24.953907 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:24.954710 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:24.955268 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:24.955435 (+ 167us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:24.958097 (+ 2662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:30:24.958107 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:30:24.958107 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:30:24.958120 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:30:24.959999 (+ 1879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:30:24.960435 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:30:24.960440 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:30:24.960440 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:30:24.960535 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:30:24.960782 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:30:24.960784 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:30:24.960784 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:30:24.961054 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:30:24.961263 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:24.961368 (+ 105us) server_negotiation.cc:299] Negotiation successful
0620 09:30:24.961476 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":87,"threads_started":1}
I20260620 09:30:24.962680 16558 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 09:30:24.964558 5717 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:30:24.965245 5717 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:30:24.969935 5717 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c2dc7f73b2a245789a258ce56fae12fe: Generated new TSK 0
I20260620 09:30:24.970116 5717 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 09:30:24.970675 5727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:24.964109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44135 (local address 127.16.43.190:37495)
0620 09:30:24.964147 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:24.964148 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:24.964414 (+ 266us) server_negotiation.cc:407] Connection header received
0620 09:30:24.964523 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:24.964524 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:24.964536 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:24.964593 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:30:24.965158 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:24.965646 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:24.966382 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:24.966599 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:24.966671 (+ 72us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:24.967997 (+ 1326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:30:24.967999 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:30:24.968000 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:30:24.968007 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:30:24.969409 (+ 1402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:30:24.969865 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:30:24.969866 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:30:24.969866 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:30:24.969927 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:30:24.970150 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:30:24.970151 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:30:24.970152 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:30:24.970280 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:30:24.970357 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:24.970559 (+ 202us) server_negotiation.cc:299] Negotiation successful
0620 09:30:24.970618 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 09:30:24.973031 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 5637
W20260620 09:30:24.978492 5622 connection.cc:535] client connection to 127.16.43.190:37495 recv error: Network error: failed to read from TLS socket (remote: 127.16.43.190:37495): Connection reset by peer (error 104)
2026-06-20T09:30:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1322 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781947562949448-16558-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:30:25 dist-test-slave-706q krb5kdc[5736](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:30:25 dist-test-slave-706q krb5kdc[5736](info): set up 2 sockets
Jun 20 09:30:25 dist-test-slave-706q krb5kdc[5736](info): commencing operation
krb5kdc: starting...
W20260620 09:30:26.025280 16558 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781947562949448-16558-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781947562949448-16558-0/krb5kdc/test-user.keytab.
Jun 20 09:30:26 dist-test-slave-706q krb5kdc[5736](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947826, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:30:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:30:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.43.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.43.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.43.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.190.keytab.
Entry for principal kudu/127.16.43.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.190.keytab.
I20260620 09:30:26.154748 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781947562949448-16558-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781947562949448-16558-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781947562949448-16558-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.43.190:38377
--webserver_interface=127.16.43.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.190.keytab
--principal=kudu/127.16.43.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.43.148:37057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.43.190:38377
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781947562949448-16558-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781947562949448-16558-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781947562949448-16558-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:30:26.234480 5749 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:30:26.234740 5749 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:30:26.234781 5749 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:30:26.236014 5749 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:30:26.236086 5749 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:30:26.236111 5749 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:30:26.236132 5749 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:30:26.236160 5749 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:30:26.237632 5749 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:37057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781947562949448-16558-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781947562949448-16558-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.43.190:38377
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.190.keytab
--principal=kudu/127.16.43.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.43.190:38377
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.43.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5749
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781947562949448-16558-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:30:26.237915 5749 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:30:26.238140 5749 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 09:30:26.243353 5749 server_base.cc:1034] running on GCE node
W20260620 09:30:26.243273 5754 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:30:26.243297 5757 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:30:26.243294 5755 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:30:26.243829 5749 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:30:26.244158 5749 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:30:26.245365 5749 hybrid_clock.cc:648] HybridClock initialized: now 1781947826245342 us; error 41 us; skew 500 ppm
Jun 20 09:30:26 dist-test-slave-706q krb5kdc[5736](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947826, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:30:26.251412 5749 init.cc:377] Logged in from keytab as kudu/127.16.43.190@KRBTEST.COM (short username kudu)
I20260620 09:30:26.252033 5749 webserver.cc:460] Webserver started at http://127.16.43.190:41197/ using document root <none> and password file <none>
I20260620 09:30:26.252306 5749 fs_manager.cc:359] Metadata directory not provided
I20260620 09:30:26.252384 5749 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:30:26.252547 5749 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:30:26.253763 5749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781947562949448-16558-0/minicluster-data/master-0/data/instance:
uuid: "78d612ca37d04d19a2ff2cbd85dc4d23"
format_stamp: "Formatted at 2026-06-20 09:30:26 on dist-test-slave-706q"
server_key: "3112d8cd927407b8794287cebaceabe1"
server_key_iv: "c3290dbd21c817613a038b91a96c1c91"
server_key_version: "encryptionkey@0"
I20260620 09:30:26.254161 5749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781947562949448-16558-0/minicluster-data/master-0/wal/instance:
uuid: "78d612ca37d04d19a2ff2cbd85dc4d23"
format_stamp: "Formatted at 2026-06-20 09:30:26 on dist-test-slave-706q"
server_key: "3112d8cd927407b8794287cebaceabe1"
server_key_iv: "c3290dbd21c817613a038b91a96c1c91"
server_key_version: "encryptionkey@0"
I20260620 09:30:26.255690 5749 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 09:30:26.256507 5764 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:30:26.256722 5749 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:30:26.256811 5749 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781947562949448-16558-0/minicluster-data/master-0/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781947562949448-16558-0/minicluster-data/master-0/wal
uuid: "78d612ca37d04d19a2ff2cbd85dc4d23"
format_stamp: "Formatted at 2026-06-20 09:30:26 on dist-test-slave-706q"
server_key: "3112d8cd927407b8794287cebaceabe1"
server_key_iv: "c3290dbd21c817613a038b91a96c1c91"
server_key_version: "encryptionkey@0"
I20260620 09:30:26.257086 5749 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781947562949448-16558-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781947562949448-16558-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781947562949448-16558-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:30:26.272516 5749 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:30:26.274686 5749 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:30:26.274817 5749 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:30:26.278692 5749 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.190:38377
I20260620 09:30:26.278723 5816 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.190:38377 every 8 connection(s)
I20260620 09:30:26.279100 5749 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
I20260620 09:30:26.279873 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 5749
I20260620 09:30:26.279971 16558 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781947562949448-16558-0/minicluster-data/master-0/wal/instance
I20260620 09:30:26.280192 16558 external_mini_cluster.cc:1428] Setting key 1b38f2e7b85e2d925368ade490e481cb
I20260620 09:30:26.280359 5817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:30:26.283926 5817 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78d612ca37d04d19a2ff2cbd85dc4d23: Bootstrap starting.
I20260620 09:30:26.284634 5817 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 78d612ca37d04d19a2ff2cbd85dc4d23: Neither blocks nor log segments found. Creating new log.
I20260620 09:30:26.284977 5817 log.cc:824] T 00000000000000000000000000000000 P 78d612ca37d04d19a2ff2cbd85dc4d23: Log is configured to *not* fsync() on all Append() calls
I20260620 09:30:26.285868 5817 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78d612ca37d04d19a2ff2cbd85dc4d23: No bootstrap required, opened a new log
I20260620 09:30:26.286221 5820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:26.280735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48883 (local address 127.16.43.190:38377)
0620 09:30:26.280927 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:26.280936 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:26.280969 (+ 33us) server_negotiation.cc:407] Connection header received
0620 09:30:26.281496 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:26.281502 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:26.281548 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:26.281672 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:30:26.281939 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:26.282847 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:26.284056 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:26.284488 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:26.284577 (+ 89us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:26.285578 (+ 1001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:30:26.285583 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:30:26.285583 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:30:26.285750 (+ 167us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:30:26.285790 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:30:26.285801 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:30:26.285869 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:26.286022 (+ 153us) server_negotiation.cc:299] Negotiation successful
0620 09:30:26.286057 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":111,"threads_started":1}
I20260620 09:30:26.287297 16558 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 09:30:26.288319 5817 raft_consensus.cc:348] T 00000000000000000000000000000000 P 78d612ca37d04d19a2ff2cbd85dc4d23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78d612ca37d04d19a2ff2cbd85dc4d23" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 38377 } }
I20260620 09:30:26.288542 5817 raft_consensus.cc:374] T 00000000000000000000000000000000 P 78d612ca37d04d19a2ff2cbd85dc4d23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:30:26.288601 5817 raft_consensus.cc:729] T 00000000000000000000000000000000 P 78d612ca37d04d19a2ff2cbd85dc4d23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78d612ca37d04d19a2ff2cbd85dc4d23, State: Initialized, Role: FOLLOWER
I20260620 09:30:26.288743 5817 consensus_queue.cc:260] T 00000000000000000000000000000000 P 78d612ca37d04d19a2ff2cbd85dc4d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78d612ca37d04d19a2ff2cbd85dc4d23" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 38377 } }
I20260620 09:30:26.288841 5817 raft_consensus.cc:388] T 00000000000000000000000000000000 P 78d612ca37d04d19a2ff2cbd85dc4d23 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:30:26.288879 5817 raft_consensus.cc:482] T 00000000000000000000000000000000 P 78d612ca37d04d19a2ff2cbd85dc4d23 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:30:26.288940 5817 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 78d612ca37d04d19a2ff2cbd85dc4d23 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:30:26.289639 5817 raft_consensus.cc:504] T 00000000000000000000000000000000 P 78d612ca37d04d19a2ff2cbd85dc4d23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78d612ca37d04d19a2ff2cbd85dc4d23" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 38377 } }
I20260620 09:30:26.289737 5817 leader_election.cc:302] T 00000000000000000000000000000000 P 78d612ca37d04d19a2ff2cbd85dc4d23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 78d612ca37d04d19a2ff2cbd85dc4d23; no voters:
I20260620 09:30:26.289932 5817 leader_election.cc:288] T 00000000000000000000000000000000 P 78d612ca37d04d19a2ff2cbd85dc4d23 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:30:26.290006 5827 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 78d612ca37d04d19a2ff2cbd85dc4d23 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:30:26.290177 5817 sys_catalog.cc:564] T 00000000000000000000000000000000 P 78d612ca37d04d19a2ff2cbd85dc4d23 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:30:26.290187 5827 raft_consensus.cc:686] T 00000000000000000000000000000000 P 78d612ca37d04d19a2ff2cbd85dc4d23 [term 1 LEADER]: Becoming Leader. State: Replica: 78d612ca37d04d19a2ff2cbd85dc4d23, State: Running, Role: LEADER
I20260620 09:30:26.290390 5827 consensus_queue.cc:237] T 00000000000000000000000000000000 P 78d612ca37d04d19a2ff2cbd85dc4d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78d612ca37d04d19a2ff2cbd85dc4d23" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 38377 } }
I20260620 09:30:26.290766 5827 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78d612ca37d04d19a2ff2cbd85dc4d23 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 78d612ca37d04d19a2ff2cbd85dc4d23. Latest consensus state: current_term: 1 leader_uuid: "78d612ca37d04d19a2ff2cbd85dc4d23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78d612ca37d04d19a2ff2cbd85dc4d23" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 38377 } } }
I20260620 09:30:26.290983 5827 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78d612ca37d04d19a2ff2cbd85dc4d23 [sys.catalog]: This master's current role is: LEADER
I20260620 09:30:26.290912 5828 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78d612ca37d04d19a2ff2cbd85dc4d23 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "78d612ca37d04d19a2ff2cbd85dc4d23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78d612ca37d04d19a2ff2cbd85dc4d23" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 38377 } } }
I20260620 09:30:26.291376 5828 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78d612ca37d04d19a2ff2cbd85dc4d23 [sys.catalog]: This master's current role is: LEADER
I20260620 09:30:26.291860 5836 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:30:26.292348 5820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:26.288750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40195 (local address 127.16.43.190:38377)
0620 09:30:26.288807 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:26.288809 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:26.288936 (+ 127us) server_negotiation.cc:407] Connection header received
0620 09:30:26.289026 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:26.289027 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:26.289038 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:26.289102 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:30:26.289264 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:26.289727 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:26.290798 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:26.291129 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:26.291171 (+ 42us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:26.292047 (+ 876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:30:26.292048 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:30:26.292048 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:30:26.292052 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:30:26.292066 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:30:26.292070 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:30:26.292121 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:26.292261 (+ 140us) server_negotiation.cc:299] Negotiation successful
0620 09:30:26.292276 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 09:30:26.292380 5836 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:30:26.293893 5836 catalog_manager.cc:1347] Generated new cluster ID: c8cdfcb463834267924d21bf619e84b2
I20260620 09:30:26.293970 5836 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:30:26.298599 5836 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:30:26.299202 5836 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:30:26.302238 5836 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 78d612ca37d04d19a2ff2cbd85dc4d23: Generated new TSK 0
I20260620 09:30:26.302453 5836 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 09:30:26.312198 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 5749
W20260620 09:30:26.316833 5734 connection.cc:535] client connection to 127.16.43.190:38377 recv error: Network error: failed to read from TLS socket (remote: 127.16.43.190:38377): Connection reset by peer (error 104)
2026-06-20T09:30:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1338 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781947562949448-16558-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:30:26 dist-test-slave-706q krb5kdc[5848](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:30:26 dist-test-slave-706q krb5kdc[5848](info): set up 2 sockets
Jun 20 09:30:26 dist-test-slave-706q krb5kdc[5848](info): commencing operation
krb5kdc: starting...
W20260620 09:30:27.352837 16558 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781947562949448-16558-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781947562949448-16558-0/krb5kdc/test-user.keytab.
Jun 20 09:30:27 dist-test-slave-706q krb5kdc[5848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947827, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:30:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:30:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.43.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.43.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.43.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.190.keytab.
Entry for principal kudu/127.16.43.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.190.keytab.
I20260620 09:30:27.479115 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781947562949448-16558-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781947562949448-16558-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781947562949448-16558-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.43.190:43703
--webserver_interface=127.16.43.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.190.keytab
--principal=kudu/127.16.43.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.43.148:36941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.43.190:43703
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781947562949448-16558-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781947562949448-16558-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781947562949448-16558-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:30:27.560468 5861 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:30:27.560675 5861 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:30:27.560699 5861 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:30:27.561776 5861 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:30:27.561810 5861 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:30:27.561823 5861 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:30:27.561836 5861 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:30:27.561847 5861 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:30:27.563390 5861 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:36941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781947562949448-16558-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781947562949448-16558-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.43.190:43703
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.190.keytab
--principal=kudu/127.16.43.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.43.190:43703
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.43.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5861
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781947562949448-16558-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:30:27.563699 5861 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:30:27.563882 5861 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:30:27.568631 5867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:30:27.568787 5861 server_base.cc:1034] running on GCE node
W20260620 09:30:27.568708 5869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:30:27.568636 5866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:30:27.569155 5861 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:30:27.569468 5861 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:30:27.570644 5861 hybrid_clock.cc:648] HybridClock initialized: now 1781947827570636 us; error 32 us; skew 500 ppm
Jun 20 09:30:27 dist-test-slave-706q krb5kdc[5848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947827, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:30:27.573189 5861 init.cc:377] Logged in from keytab as kudu/127.16.43.190@KRBTEST.COM (short username kudu)
I20260620 09:30:27.573673 5861 webserver.cc:460] Webserver started at http://127.16.43.190:40057/ using document root <none> and password file <none>
I20260620 09:30:27.573900 5861 fs_manager.cc:359] Metadata directory not provided
I20260620 09:30:27.573971 5861 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:30:27.574119 5861 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:30:27.575188 5861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781947562949448-16558-0/minicluster-data/master-0/data/instance:
uuid: "14ae7bf88fd4437087d3e7faeebb6238"
format_stamp: "Formatted at 2026-06-20 09:30:27 on dist-test-slave-706q"
server_key: "6d5f574354e0bba8f33cba52b8755250"
server_key_iv: "e00133e6eb6a1e3852dc6014bd8989d4"
server_key_version: "encryptionkey@0"
I20260620 09:30:27.575604 5861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781947562949448-16558-0/minicluster-data/master-0/wal/instance:
uuid: "14ae7bf88fd4437087d3e7faeebb6238"
format_stamp: "Formatted at 2026-06-20 09:30:27 on dist-test-slave-706q"
server_key: "6d5f574354e0bba8f33cba52b8755250"
server_key_iv: "e00133e6eb6a1e3852dc6014bd8989d4"
server_key_version: "encryptionkey@0"
I20260620 09:30:27.576972 5861 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 09:30:27.577720 5876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:30:27.577886 5861 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 09:30:27.577986 5861 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781947562949448-16558-0/minicluster-data/master-0/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781947562949448-16558-0/minicluster-data/master-0/wal
uuid: "14ae7bf88fd4437087d3e7faeebb6238"
format_stamp: "Formatted at 2026-06-20 09:30:27 on dist-test-slave-706q"
server_key: "6d5f574354e0bba8f33cba52b8755250"
server_key_iv: "e00133e6eb6a1e3852dc6014bd8989d4"
server_key_version: "encryptionkey@0"
I20260620 09:30:27.578073 5861 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781947562949448-16558-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781947562949448-16558-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781947562949448-16558-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:30:27.581394 5861 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:30:27.583237 5861 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:30:27.583396 5861 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:30:27.587260 5861 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.190:43703
I20260620 09:30:27.587288 5928 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.190:43703 every 8 connection(s)
I20260620 09:30:27.587690 5861 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
I20260620 09:30:27.588678 5929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:30:27.590914 5929 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14ae7bf88fd4437087d3e7faeebb6238: Bootstrap starting.
I20260620 09:30:27.591488 5929 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 14ae7bf88fd4437087d3e7faeebb6238: Neither blocks nor log segments found. Creating new log.
I20260620 09:30:27.591828 5929 log.cc:824] T 00000000000000000000000000000000 P 14ae7bf88fd4437087d3e7faeebb6238: Log is configured to *not* fsync() on all Append() calls
I20260620 09:30:27.592515 5929 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14ae7bf88fd4437087d3e7faeebb6238: No bootstrap required, opened a new log
I20260620 09:30:27.594136 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 5861
I20260620 09:30:27.594257 16558 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781947562949448-16558-0/minicluster-data/master-0/wal/instance
I20260620 09:30:27.594462 16558 external_mini_cluster.cc:1428] Setting key 47757d697eca9182d9169078925f787a
I20260620 09:30:27.594328 5929 raft_consensus.cc:348] T 00000000000000000000000000000000 P 14ae7bf88fd4437087d3e7faeebb6238 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14ae7bf88fd4437087d3e7faeebb6238" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 43703 } }
I20260620 09:30:27.594508 5929 raft_consensus.cc:374] T 00000000000000000000000000000000 P 14ae7bf88fd4437087d3e7faeebb6238 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:30:27.594548 5929 raft_consensus.cc:729] T 00000000000000000000000000000000 P 14ae7bf88fd4437087d3e7faeebb6238 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14ae7bf88fd4437087d3e7faeebb6238, State: Initialized, Role: FOLLOWER
I20260620 09:30:27.594697 5929 consensus_queue.cc:260] T 00000000000000000000000000000000 P 14ae7bf88fd4437087d3e7faeebb6238 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14ae7bf88fd4437087d3e7faeebb6238" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 43703 } }
I20260620 09:30:27.594789 5929 raft_consensus.cc:388] T 00000000000000000000000000000000 P 14ae7bf88fd4437087d3e7faeebb6238 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:30:27.594846 5929 raft_consensus.cc:482] T 00000000000000000000000000000000 P 14ae7bf88fd4437087d3e7faeebb6238 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:30:27.594901 5929 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 14ae7bf88fd4437087d3e7faeebb6238 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:30:27.595916 5929 raft_consensus.cc:504] T 00000000000000000000000000000000 P 14ae7bf88fd4437087d3e7faeebb6238 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14ae7bf88fd4437087d3e7faeebb6238" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 43703 } }
I20260620 09:30:27.596009 5929 leader_election.cc:302] T 00000000000000000000000000000000 P 14ae7bf88fd4437087d3e7faeebb6238 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 14ae7bf88fd4437087d3e7faeebb6238; no voters:
I20260620 09:30:27.596206 5929 leader_election.cc:288] T 00000000000000000000000000000000 P 14ae7bf88fd4437087d3e7faeebb6238 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:30:27.596235 5934 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 14ae7bf88fd4437087d3e7faeebb6238 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:30:27.596351 5934 raft_consensus.cc:686] T 00000000000000000000000000000000 P 14ae7bf88fd4437087d3e7faeebb6238 [term 1 LEADER]: Becoming Leader. State: Replica: 14ae7bf88fd4437087d3e7faeebb6238, State: Running, Role: LEADER
I20260620 09:30:27.596472 5929 sys_catalog.cc:564] T 00000000000000000000000000000000 P 14ae7bf88fd4437087d3e7faeebb6238 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:30:27.596443 5934 consensus_queue.cc:237] T 00000000000000000000000000000000 P 14ae7bf88fd4437087d3e7faeebb6238 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14ae7bf88fd4437087d3e7faeebb6238" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 43703 } }
I20260620 09:30:27.596772 5934 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14ae7bf88fd4437087d3e7faeebb6238 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "14ae7bf88fd4437087d3e7faeebb6238" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14ae7bf88fd4437087d3e7faeebb6238" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 43703 } } }
I20260620 09:30:27.596840 5933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:27.595122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53729 (local address 127.16.43.190:43703)
0620 09:30:27.595312 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:27.595322 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:27.595357 (+ 35us) server_negotiation.cc:407] Connection header received
0620 09:30:27.595457 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:27.595463 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:27.595486 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:27.595647 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:30:27.596394 (+ 747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:30:27.596398 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:30:27.596398 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:30:27.596403 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:30:27.596452 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:30:27.596466 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:30:27.596525 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:27.596596 (+ 71us) server_negotiation.cc:299] Negotiation successful
0620 09:30:27.596636 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":103,"threads_started":1}
I20260620 09:30:27.596843 5934 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14ae7bf88fd4437087d3e7faeebb6238 [sys.catalog]: This master's current role is: LEADER
I20260620 09:30:27.597121 5934 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14ae7bf88fd4437087d3e7faeebb6238 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 14ae7bf88fd4437087d3e7faeebb6238. Latest consensus state: current_term: 1 leader_uuid: "14ae7bf88fd4437087d3e7faeebb6238" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14ae7bf88fd4437087d3e7faeebb6238" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 43703 } } }
I20260620 09:30:27.597195 5934 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14ae7bf88fd4437087d3e7faeebb6238 [sys.catalog]: This master's current role is: LEADER
W20260620 09:30:27.598420 5950 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 14ae7bf88fd4437087d3e7faeebb6238: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 09:30:27.598501 5950 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 09:30:27.598534 16558 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 09:30:27.598584 5941 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:30:27.598752 5941 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:30:27.600356 5941 catalog_manager.cc:1347] Generated new cluster ID: d0cd68c594ed40eaacb81d2fd4caf233
I20260620 09:30:27.600407 5941 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:30:27.601332 5933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:27.600071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50225 (local address 127.16.43.190:43703)
0620 09:30:27.600103 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:27.600105 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:27.600245 (+ 140us) server_negotiation.cc:407] Connection header received
0620 09:30:27.600338 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:27.600339 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:27.600344 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:27.600373 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:30:27.601078 (+ 705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:30:27.601079 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:30:27.601079 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:30:27.601084 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:30:27.601098 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:30:27.601102 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:30:27.601136 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:27.601258 (+ 122us) server_negotiation.cc:299] Negotiation successful
0620 09:30:27.601272 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260620 09:30:27.618651 5941 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:30:27.618857 5941 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:30:27.624673 5941 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 14ae7bf88fd4437087d3e7faeebb6238: Generated new TSK 0
I20260620 09:30:27.624907 5941 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 09:30:27.628154 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 5861
2026-06-20T09:30:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1315 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781947562949448-16558-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:30:27 dist-test-slave-706q krb5kdc[5960](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:30:27 dist-test-slave-706q krb5kdc[5960](info): set up 2 sockets
Jun 20 09:30:27 dist-test-slave-706q krb5kdc[5960](info): commencing operation
krb5kdc: starting...
W20260620 09:30:28.672675 16558 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781947562949448-16558-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781947562949448-16558-0/krb5kdc/test-user.keytab.
Jun 20 09:30:28 dist-test-slave-706q krb5kdc[5960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947828, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:30:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:30:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.43.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.43.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.43.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.190.keytab.
Entry for principal kudu/127.16.43.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.190.keytab.
I20260620 09:30:28.795445 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781947562949448-16558-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781947562949448-16558-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781947562949448-16558-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.43.190:35737
--webserver_interface=127.16.43.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.190.keytab
--principal=kudu/127.16.43.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.43.148:41931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.43.190:35737
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781947562949448-16558-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781947562949448-16558-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781947562949448-16558-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:30:28.879274 5973 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:30:28.879520 5973 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:30:28.879585 5973 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:30:28.880792 5973 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:30:28.880867 5973 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:30:28.880893 5973 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:30:28.880925 5973 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:30:28.880952 5973 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:30:28.882442 5973 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:41931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781947562949448-16558-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781947562949448-16558-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.43.190:35737
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.190.keytab
--principal=kudu/127.16.43.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.43.190:35737
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.43.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5973
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781947562949448-16558-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:30:28.882719 5973 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:30:28.882952 5973 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:30:28.888010 5979 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:30:28.888088 5978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:30:28.888067 5981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:30:28.888072 5973 server_base.cc:1034] running on GCE node
I20260620 09:30:28.888563 5973 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:30:28.888865 5973 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:30:28.890004 5973 hybrid_clock.cc:648] HybridClock initialized: now 1781947828889986 us; error 38 us; skew 500 ppm
Jun 20 09:30:28 dist-test-slave-706q krb5kdc[5960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947828, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:30:28.892510 5973 init.cc:377] Logged in from keytab as kudu/127.16.43.190@KRBTEST.COM (short username kudu)
I20260620 09:30:28.892886 5973 webserver.cc:460] Webserver started at http://127.16.43.190:34649/ using document root <none> and password file <none>
I20260620 09:30:28.893075 5973 fs_manager.cc:359] Metadata directory not provided
I20260620 09:30:28.893114 5973 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:30:28.893220 5973 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:30:28.894261 5973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781947562949448-16558-0/minicluster-data/master-0/data/instance:
uuid: "1d05b1ec38064796b4bdc785d871c8ac"
format_stamp: "Formatted at 2026-06-20 09:30:28 on dist-test-slave-706q"
server_key: "ea27df9b9a12c191c8f4c3d55b8d2ea1"
server_key_iv: "9df0c1669d63c9a2826885ef36653182"
server_key_version: "encryptionkey@0"
I20260620 09:30:28.894598 5973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781947562949448-16558-0/minicluster-data/master-0/wal/instance:
uuid: "1d05b1ec38064796b4bdc785d871c8ac"
format_stamp: "Formatted at 2026-06-20 09:30:28 on dist-test-slave-706q"
server_key: "ea27df9b9a12c191c8f4c3d55b8d2ea1"
server_key_iv: "9df0c1669d63c9a2826885ef36653182"
server_key_version: "encryptionkey@0"
I20260620 09:30:28.895960 5973 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 09:30:28.896664 5988 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:30:28.896835 5973 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 09:30:28.896927 5973 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781947562949448-16558-0/minicluster-data/master-0/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781947562949448-16558-0/minicluster-data/master-0/wal
uuid: "1d05b1ec38064796b4bdc785d871c8ac"
format_stamp: "Formatted at 2026-06-20 09:30:28 on dist-test-slave-706q"
server_key: "ea27df9b9a12c191c8f4c3d55b8d2ea1"
server_key_iv: "9df0c1669d63c9a2826885ef36653182"
server_key_version: "encryptionkey@0"
I20260620 09:30:28.897011 5973 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781947562949448-16558-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781947562949448-16558-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781947562949448-16558-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:30:28.920197 5973 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:30:28.922549 5973 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:30:28.922720 5973 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:30:28.926725 5973 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.190:35737
I20260620 09:30:28.926805 6040 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.190:35737 every 8 connection(s)
I20260620 09:30:28.927150 5973 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
I20260620 09:30:28.928030 6041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:30:28.930219 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 5973
I20260620 09:30:28.930343 16558 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781947562949448-16558-0/minicluster-data/master-0/wal/instance
I20260620 09:30:28.930542 16558 external_mini_cluster.cc:1428] Setting key c00df5b1b038ebbbe2dee9ff71a7048b
I20260620 09:30:28.930521 6041 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d05b1ec38064796b4bdc785d871c8ac: Bootstrap starting.
I20260620 09:30:28.931208 6041 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1d05b1ec38064796b4bdc785d871c8ac: Neither blocks nor log segments found. Creating new log.
I20260620 09:30:28.931504 6041 log.cc:824] T 00000000000000000000000000000000 P 1d05b1ec38064796b4bdc785d871c8ac: Log is configured to *not* fsync() on all Append() calls
I20260620 09:30:28.932224 6041 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d05b1ec38064796b4bdc785d871c8ac: No bootstrap required, opened a new log
I20260620 09:30:28.934010 6041 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1d05b1ec38064796b4bdc785d871c8ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d05b1ec38064796b4bdc785d871c8ac" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 35737 } }
I20260620 09:30:28.934194 6041 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1d05b1ec38064796b4bdc785d871c8ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:30:28.934271 6041 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1d05b1ec38064796b4bdc785d871c8ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d05b1ec38064796b4bdc785d871c8ac, State: Initialized, Role: FOLLOWER
I20260620 09:30:28.934444 6041 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d05b1ec38064796b4bdc785d871c8ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d05b1ec38064796b4bdc785d871c8ac" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 35737 } }
I20260620 09:30:28.934542 6041 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1d05b1ec38064796b4bdc785d871c8ac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:30:28.934603 6041 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1d05b1ec38064796b4bdc785d871c8ac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:30:28.934657 6041 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1d05b1ec38064796b4bdc785d871c8ac [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:30:28.935199 6041 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1d05b1ec38064796b4bdc785d871c8ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d05b1ec38064796b4bdc785d871c8ac" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 35737 } }
I20260620 09:30:28.935329 6041 leader_election.cc:302] T 00000000000000000000000000000000 P 1d05b1ec38064796b4bdc785d871c8ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1d05b1ec38064796b4bdc785d871c8ac; no voters:
I20260620 09:30:28.935552 6041 leader_election.cc:288] T 00000000000000000000000000000000 P 1d05b1ec38064796b4bdc785d871c8ac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:30:28.935597 6046 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1d05b1ec38064796b4bdc785d871c8ac [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:30:28.935743 6046 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1d05b1ec38064796b4bdc785d871c8ac [term 1 LEADER]: Becoming Leader. State: Replica: 1d05b1ec38064796b4bdc785d871c8ac, State: Running, Role: LEADER
I20260620 09:30:28.935815 6041 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1d05b1ec38064796b4bdc785d871c8ac [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:30:28.935897 6046 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1d05b1ec38064796b4bdc785d871c8ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d05b1ec38064796b4bdc785d871c8ac" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 35737 } }
I20260620 09:30:28.936323 6047 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d05b1ec38064796b4bdc785d871c8ac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1d05b1ec38064796b4bdc785d871c8ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d05b1ec38064796b4bdc785d871c8ac" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 35737 } } }
I20260620 09:30:28.936450 6047 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d05b1ec38064796b4bdc785d871c8ac [sys.catalog]: This master's current role is: LEADER
I20260620 09:30:28.936717 6055 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:30:28.937414 6055 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:30:28.937584 6048 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d05b1ec38064796b4bdc785d871c8ac [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d05b1ec38064796b4bdc785d871c8ac. Latest consensus state: current_term: 1 leader_uuid: "1d05b1ec38064796b4bdc785d871c8ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d05b1ec38064796b4bdc785d871c8ac" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 35737 } } }
I20260620 09:30:28.937665 6048 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d05b1ec38064796b4bdc785d871c8ac [sys.catalog]: This master's current role is: LEADER
Jun 20 09:30:28 dist-test-slave-706q krb5kdc[5960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947828, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.43.190@KRBTEST.COM
I20260620 09:30:28.939790 6055 catalog_manager.cc:1347] Generated new cluster ID: c197162f9e184938b6af5c72a72e252a
I20260620 09:30:28.939844 6055 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:30:28.942286 6044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:28.931106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45895 (local address 127.16.43.190:35737)
0620 09:30:28.931296 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:28.931306 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:28.931340 (+ 34us) server_negotiation.cc:407] Connection header received
0620 09:30:28.931885 (+ 545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:28.931892 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:28.931939 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:28.932093 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:30:28.932839 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:28.933712 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:28.934600 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:28.935076 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:28.935187 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:28.938556 (+ 3369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:30:28.938561 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:30:28.938561 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:30:28.938574 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:30:28.940535 (+ 1961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:30:28.940976 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:30:28.940980 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:30:28.940980 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:30:28.941097 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:30:28.941349 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:30:28.941351 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:30:28.941351 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:30:28.941622 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:30:28.941802 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:28.941927 (+ 125us) server_negotiation.cc:299] Negotiation successful
0620 09:30:28.942104 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":82,"threads_started":1}
I20260620 09:30:28.943396 16558 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 09:30:28.952116 6044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:28.944930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58883 (local address 127.16.43.190:35737)
0620 09:30:28.944973 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:28.944975 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:28.945137 (+ 162us) server_negotiation.cc:407] Connection header received
0620 09:30:28.945209 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:28.945210 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:28.945221 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:28.945282 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:30:28.945933 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:28.946494 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:28.947322 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:28.947680 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:28.947773 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:28.949162 (+ 1389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:30:28.949163 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:30:28.949164 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:30:28.949171 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:30:28.950746 (+ 1575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:30:28.951153 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:30:28.951154 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:30:28.951155 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:30:28.951218 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:30:28.951439 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:30:28.951440 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:30:28.951441 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:30:28.951640 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:30:28.951763 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:28.951912 (+ 149us) server_negotiation.cc:299] Negotiation successful
0620 09:30:28.952007 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 09:30:28.952632 6055 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:30:28.953382 6055 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:30:28.965648 6055 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1d05b1ec38064796b4bdc785d871c8ac: Generated new TSK 0
I20260620 09:30:28.965982 6055 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 09:30:28.976464 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 5973
2026-06-20T09:30:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1348 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781947562949448-16558-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:30:29 dist-test-slave-706q krb5kdc[6072](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:30:29 dist-test-slave-706q krb5kdc[6072](info): set up 2 sockets
Jun 20 09:30:29 dist-test-slave-706q krb5kdc[6072](info): commencing operation
krb5kdc: starting...
W20260620 09:30:30.036906 16558 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781947562949448-16558-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781947562949448-16558-0/krb5kdc/test-user.keytab.
Jun 20 09:30:30 dist-test-slave-706q krb5kdc[6072](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947830, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:30:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:30:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.43.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.43.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.43.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.190.keytab.
Entry for principal kudu/127.16.43.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.190.keytab.
I20260620 09:30:30.162022 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781947562949448-16558-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781947562949448-16558-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781947562949448-16558-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.43.190:33713
--webserver_interface=127.16.43.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.190.keytab
--principal=kudu/127.16.43.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.43.148:45731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.43.190:33713
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781947562949448-16558-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781947562949448-16558-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781947562949448-16558-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:30:30.244954 6085 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:30:30.245189 6085 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:30:30.245215 6085 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:30:30.246516 6085 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:30:30.246562 6085 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:30:30.246577 6085 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:30:30.246590 6085 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:30:30.246603 6085 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:30:30.248188 6085 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:45731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781947562949448-16558-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781947562949448-16558-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.43.190:33713
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.190.keytab
--principal=kudu/127.16.43.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.43.190:33713
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.43.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6085
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781947562949448-16558-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:30:30.248472 6085 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:30:30.248719 6085 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:30:30.253789 6093 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:30:30.253788 6091 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:30:30.253876 6090 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:30:30.254033 6085 server_base.cc:1034] running on GCE node
I20260620 09:30:30.254272 6085 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:30:30.254645 6085 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:30:30.255820 6085 hybrid_clock.cc:648] HybridClock initialized: now 1781947830255793 us; error 37 us; skew 500 ppm
Jun 20 09:30:30 dist-test-slave-706q krb5kdc[6072](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947830, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:30:30.258430 6085 init.cc:377] Logged in from keytab as kudu/127.16.43.190@KRBTEST.COM (short username kudu)
I20260620 09:30:30.259038 6085 webserver.cc:460] Webserver started at http://127.16.43.190:39271/ using document root <none> and password file <none>
I20260620 09:30:30.259301 6085 fs_manager.cc:359] Metadata directory not provided
I20260620 09:30:30.259378 6085 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:30:30.259599 6085 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:30:30.260730 6085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781947562949448-16558-0/minicluster-data/master-0/data/instance:
uuid: "aad75f9b39754729ad893e05331ad812"
format_stamp: "Formatted at 2026-06-20 09:30:30 on dist-test-slave-706q"
server_key: "f49cf73d5816b4eed62ca7c5fbd11a90"
server_key_iv: "7203ad817a8b388818674026f322cd6a"
server_key_version: "encryptionkey@0"
I20260620 09:30:30.261135 6085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781947562949448-16558-0/minicluster-data/master-0/wal/instance:
uuid: "aad75f9b39754729ad893e05331ad812"
format_stamp: "Formatted at 2026-06-20 09:30:30 on dist-test-slave-706q"
server_key: "f49cf73d5816b4eed62ca7c5fbd11a90"
server_key_iv: "7203ad817a8b388818674026f322cd6a"
server_key_version: "encryptionkey@0"
I20260620 09:30:30.262661 6085 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 09:30:30.263491 6100 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:30:30.263762 6085 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 09:30:30.263871 6085 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781947562949448-16558-0/minicluster-data/master-0/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781947562949448-16558-0/minicluster-data/master-0/wal
uuid: "aad75f9b39754729ad893e05331ad812"
format_stamp: "Formatted at 2026-06-20 09:30:30 on dist-test-slave-706q"
server_key: "f49cf73d5816b4eed62ca7c5fbd11a90"
server_key_iv: "7203ad817a8b388818674026f322cd6a"
server_key_version: "encryptionkey@0"
I20260620 09:30:30.263962 6085 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781947562949448-16558-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781947562949448-16558-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781947562949448-16558-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:30:30.283018 6085 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:30:30.285219 6085 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:30:30.285444 6085 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:30:30.290174 6085 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.190:33713
I20260620 09:30:30.290206 6152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.190:33713 every 8 connection(s)
I20260620 09:30:30.290575 6085 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
I20260620 09:30:30.291504 6153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:30:30.293844 6153 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aad75f9b39754729ad893e05331ad812: Bootstrap starting.
I20260620 09:30:30.294406 6153 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aad75f9b39754729ad893e05331ad812: Neither blocks nor log segments found. Creating new log.
I20260620 09:30:30.294646 6153 log.cc:824] T 00000000000000000000000000000000 P aad75f9b39754729ad893e05331ad812: Log is configured to *not* fsync() on all Append() calls
I20260620 09:30:30.295439 6153 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aad75f9b39754729ad893e05331ad812: No bootstrap required, opened a new log
I20260620 09:30:30.297219 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 6085
I20260620 09:30:30.297186 6153 raft_consensus.cc:348] T 00000000000000000000000000000000 P aad75f9b39754729ad893e05331ad812 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aad75f9b39754729ad893e05331ad812" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 33713 } }
I20260620 09:30:30.297350 16558 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781947562949448-16558-0/minicluster-data/master-0/wal/instance
I20260620 09:30:30.297319 6153 raft_consensus.cc:374] T 00000000000000000000000000000000 P aad75f9b39754729ad893e05331ad812 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:30:30.297389 6153 raft_consensus.cc:729] T 00000000000000000000000000000000 P aad75f9b39754729ad893e05331ad812 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aad75f9b39754729ad893e05331ad812, State: Initialized, Role: FOLLOWER
I20260620 09:30:30.297544 6153 consensus_queue.cc:260] T 00000000000000000000000000000000 P aad75f9b39754729ad893e05331ad812 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aad75f9b39754729ad893e05331ad812" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 33713 } }
I20260620 09:30:30.297607 6153 raft_consensus.cc:388] T 00000000000000000000000000000000 P aad75f9b39754729ad893e05331ad812 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:30:30.297648 16558 external_mini_cluster.cc:1428] Setting key deb6dd17723c9ec4fc068defd1fb30ba
I20260620 09:30:30.297660 6153 raft_consensus.cc:482] T 00000000000000000000000000000000 P aad75f9b39754729ad893e05331ad812 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:30:30.297708 6153 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aad75f9b39754729ad893e05331ad812 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:30:30.298904 6153 raft_consensus.cc:504] T 00000000000000000000000000000000 P aad75f9b39754729ad893e05331ad812 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aad75f9b39754729ad893e05331ad812" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 33713 } }
I20260620 09:30:30.299017 6153 leader_election.cc:302] T 00000000000000000000000000000000 P aad75f9b39754729ad893e05331ad812 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aad75f9b39754729ad893e05331ad812; no voters:
I20260620 09:30:30.299180 6153 leader_election.cc:288] T 00000000000000000000000000000000 P aad75f9b39754729ad893e05331ad812 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:30:30.299206 6158 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aad75f9b39754729ad893e05331ad812 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:30:30.299309 6158 raft_consensus.cc:686] T 00000000000000000000000000000000 P aad75f9b39754729ad893e05331ad812 [term 1 LEADER]: Becoming Leader. State: Replica: aad75f9b39754729ad893e05331ad812, State: Running, Role: LEADER
I20260620 09:30:30.299391 6158 consensus_queue.cc:237] T 00000000000000000000000000000000 P aad75f9b39754729ad893e05331ad812 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aad75f9b39754729ad893e05331ad812" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 33713 } }
I20260620 09:30:30.299449 6153 sys_catalog.cc:564] T 00000000000000000000000000000000 P aad75f9b39754729ad893e05331ad812 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:30:30.299772 6159 sys_catalog.cc:455] T 00000000000000000000000000000000 P aad75f9b39754729ad893e05331ad812 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aad75f9b39754729ad893e05331ad812" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aad75f9b39754729ad893e05331ad812" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 33713 } } }
I20260620 09:30:30.299882 6159 sys_catalog.cc:458] T 00000000000000000000000000000000 P aad75f9b39754729ad893e05331ad812 [sys.catalog]: This master's current role is: LEADER
I20260620 09:30:30.299765 6160 sys_catalog.cc:455] T 00000000000000000000000000000000 P aad75f9b39754729ad893e05331ad812 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aad75f9b39754729ad893e05331ad812. Latest consensus state: current_term: 1 leader_uuid: "aad75f9b39754729ad893e05331ad812" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aad75f9b39754729ad893e05331ad812" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 33713 } } }
I20260620 09:30:30.300155 6160 sys_catalog.cc:458] T 00000000000000000000000000000000 P aad75f9b39754729ad893e05331ad812 [sys.catalog]: This master's current role is: LEADER
I20260620 09:30:30.300410 6164 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:30:30.300956 6164 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:30:30.303014 6164 catalog_manager.cc:1347] Generated new cluster ID: 60694e4a3b1f4b43b8aa34dad426cdae
I20260620 09:30:30.303066 6164 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:30:30.304139 6157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:30.298186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39257 (local address 127.16.43.190:33713)
0620 09:30:30.298419 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:30.298431 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:30.298475 (+ 44us) server_negotiation.cc:407] Connection header received
0620 09:30:30.298618 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:30.298625 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:30.298691 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:30.298841 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:30:30.299136 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:30.300593 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:30.301794 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:30.302353 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:30.302497 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:30.303288 (+ 791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:30:30.303291 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:30:30.303292 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:30:30.303485 (+ 193us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:30:30.303520 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:30:30.303568 (+ 48us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:30:30.303616 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:30.303870 (+ 254us) server_negotiation.cc:299] Negotiation successful
0620 09:30:30.303932 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":102,"threads_started":1}
I20260620 09:30:30.305045 16558 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 09:30:30.309979 6157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:30.306471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37653 (local address 127.16.43.190:33713)
0620 09:30:30.306504 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:30.306505 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:30.306647 (+ 142us) server_negotiation.cc:407] Connection header received
0620 09:30:30.306703 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:30.306703 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:30.306713 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:30.306770 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:30:30.306908 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:30.307622 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:30.308675 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:30.308967 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:30.309026 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:30.309729 (+ 703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:30:30.309730 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:30:30.309731 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:30:30.309736 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:30:30.309751 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:30:30.309755 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:30:30.309807 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:30.309875 (+ 68us) server_negotiation.cc:299] Negotiation successful
0620 09:30:30.309899 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260620 09:30:30.314206 6164 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:30:30.314944 6164 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:30:30.320119 6164 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aad75f9b39754729ad893e05331ad812: Generated new TSK 0
I20260620 09:30:30.320312 6164 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 09:30:30.326838 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 6085
2026-06-20T09:30:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1350 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781947562949448-16558-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:30:30 dist-test-slave-706q krb5kdc[6184](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:30:30 dist-test-slave-706q krb5kdc[6184](info): set up 2 sockets
Jun 20 09:30:30 dist-test-slave-706q krb5kdc[6184](info): commencing operation
krb5kdc: starting...
W20260620 09:30:31.368886 16558 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781947562949448-16558-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781947562949448-16558-0/krb5kdc/test-user.keytab.
Jun 20 09:30:31 dist-test-slave-706q krb5kdc[6184](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947831, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:30:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:30:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.43.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.43.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.43.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.190.keytab.
Entry for principal kudu/127.16.43.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.190.keytab.
I20260620 09:30:31.494709 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781947562949448-16558-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781947562949448-16558-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781947562949448-16558-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.43.190:45763
--webserver_interface=127.16.43.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.190.keytab
--principal=kudu/127.16.43.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.43.148:46535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.43.190:45763
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781947562949448-16558-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781947562949448-16558-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781947562949448-16558-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:30:31.576445 6197 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:30:31.576673 6197 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:30:31.576696 6197 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:30:31.577894 6197 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:30:31.577941 6197 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:30:31.577955 6197 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:30:31.578003 6197 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:30:31.578035 6197 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:30:31.579517 6197 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:46535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781947562949448-16558-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781947562949448-16558-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.43.190:45763
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.190.keytab
--principal=kudu/127.16.43.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.43.190:45763
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.43.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6197
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781947562949448-16558-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:30:31.579751 6197 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:30:31.579999 6197 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 09:30:31.585124 6197 server_base.cc:1034] running on GCE node
W20260620 09:30:31.585052 6205 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:30:31.585022 6202 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:30:31.585016 6203 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:30:31.585633 6197 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:30:31.585954 6197 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:30:31.587142 6197 hybrid_clock.cc:648] HybridClock initialized: now 1781947831587117 us; error 61 us; skew 500 ppm
Jun 20 09:30:31 dist-test-slave-706q krb5kdc[6184](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947831, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:30:31.593010 6197 init.cc:377] Logged in from keytab as kudu/127.16.43.190@KRBTEST.COM (short username kudu)
I20260620 09:30:31.593592 6197 webserver.cc:460] Webserver started at http://127.16.43.190:38063/ using document root <none> and password file <none>
I20260620 09:30:31.593832 6197 fs_manager.cc:359] Metadata directory not provided
I20260620 09:30:31.593904 6197 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:30:31.594056 6197 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:30:31.595187 6197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781947562949448-16558-0/minicluster-data/master-0/data/instance:
uuid: "f2789df0324d453bbe12ff12d1cfee40"
format_stamp: "Formatted at 2026-06-20 09:30:31 on dist-test-slave-706q"
server_key: "ef77766b030fe8e5cad9bf40b1b23da9"
server_key_iv: "f3d57b82e6921a477f00997ec46b4045"
server_key_version: "encryptionkey@0"
I20260620 09:30:31.595642 6197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781947562949448-16558-0/minicluster-data/master-0/wal/instance:
uuid: "f2789df0324d453bbe12ff12d1cfee40"
format_stamp: "Formatted at 2026-06-20 09:30:31 on dist-test-slave-706q"
server_key: "ef77766b030fe8e5cad9bf40b1b23da9"
server_key_iv: "f3d57b82e6921a477f00997ec46b4045"
server_key_version: "encryptionkey@0"
I20260620 09:30:31.597024 6197 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 09:30:31.597832 6212 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:30:31.598027 6197 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 09:30:31.598137 6197 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781947562949448-16558-0/minicluster-data/master-0/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781947562949448-16558-0/minicluster-data/master-0/wal
uuid: "f2789df0324d453bbe12ff12d1cfee40"
format_stamp: "Formatted at 2026-06-20 09:30:31 on dist-test-slave-706q"
server_key: "ef77766b030fe8e5cad9bf40b1b23da9"
server_key_iv: "f3d57b82e6921a477f00997ec46b4045"
server_key_version: "encryptionkey@0"
I20260620 09:30:31.598227 6197 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781947562949448-16558-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781947562949448-16558-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781947562949448-16558-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:30:31.601694 6197 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:30:31.603890 6197 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:30:31.604056 6197 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:30:31.608224 6197 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.190:45763
I20260620 09:30:31.608251 6264 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.190:45763 every 8 connection(s)
I20260620 09:30:31.608594 6197 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
I20260620 09:30:31.609485 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 6197
I20260620 09:30:31.609546 6265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:30:31.609663 16558 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781947562949448-16558-0/minicluster-data/master-0/wal/instance
I20260620 09:30:31.609894 16558 external_mini_cluster.cc:1428] Setting key c55d5c412925c2cfe0f3956a9b981783
I20260620 09:30:31.612488 6268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:31.610475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57855 (local address 127.16.43.190:45763)
0620 09:30:31.610637 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:31.610646 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:31.610687 (+ 41us) server_negotiation.cc:407] Connection header received
0620 09:30:31.611142 (+ 455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:31.611149 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:31.611174 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:31.611294 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:30:31.612000 (+ 706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:30:31.612004 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:30:31.612004 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:30:31.612008 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:30:31.612058 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:30:31.612068 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:30:31.612139 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:31.612234 (+ 95us) server_negotiation.cc:299] Negotiation successful
0620 09:30:31.612294 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":82,"threads_started":1}
I20260620 09:30:31.613125 6265 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2789df0324d453bbe12ff12d1cfee40: Bootstrap starting.
I20260620 09:30:31.613740 16558 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 09:30:31.613893 6265 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f2789df0324d453bbe12ff12d1cfee40: Neither blocks nor log segments found. Creating new log.
I20260620 09:30:31.614184 6265 log.cc:824] T 00000000000000000000000000000000 P f2789df0324d453bbe12ff12d1cfee40: Log is configured to *not* fsync() on all Append() calls
I20260620 09:30:31.614924 6265 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2789df0324d453bbe12ff12d1cfee40: No bootstrap required, opened a new log
I20260620 09:30:31.616554 6268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:31.615355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50553 (local address 127.16.43.190:45763)
0620 09:30:31.615390 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:31.615392 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:31.615522 (+ 130us) server_negotiation.cc:407] Connection header received
0620 09:30:31.615656 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:31.615657 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:31.615662 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:31.615721 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:30:31.616348 (+ 627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:30:31.616349 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:30:31.616350 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:30:31.616354 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:30:31.616369 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:30:31.616373 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:30:31.616423 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:31.616471 (+ 48us) server_negotiation.cc:299] Negotiation successful
0620 09:30:31.616485 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260620 09:30:31.617168 6265 raft_consensus.cc:348] T 00000000000000000000000000000000 P f2789df0324d453bbe12ff12d1cfee40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2789df0324d453bbe12ff12d1cfee40" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 45763 } }
I20260620 09:30:31.617311 6265 raft_consensus.cc:374] T 00000000000000000000000000000000 P f2789df0324d453bbe12ff12d1cfee40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:30:31.617332 6265 raft_consensus.cc:729] T 00000000000000000000000000000000 P f2789df0324d453bbe12ff12d1cfee40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2789df0324d453bbe12ff12d1cfee40, State: Initialized, Role: FOLLOWER
I20260620 09:30:31.617479 6265 consensus_queue.cc:260] T 00000000000000000000000000000000 P f2789df0324d453bbe12ff12d1cfee40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2789df0324d453bbe12ff12d1cfee40" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 45763 } }
I20260620 09:30:31.617607 6265 raft_consensus.cc:388] T 00000000000000000000000000000000 P f2789df0324d453bbe12ff12d1cfee40 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:30:31.617643 6265 raft_consensus.cc:482] T 00000000000000000000000000000000 P f2789df0324d453bbe12ff12d1cfee40 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:30:31.617719 6265 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f2789df0324d453bbe12ff12d1cfee40 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:30:31.618391 6265 raft_consensus.cc:504] T 00000000000000000000000000000000 P f2789df0324d453bbe12ff12d1cfee40 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2789df0324d453bbe12ff12d1cfee40" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 45763 } }
I20260620 09:30:31.618480 6265 leader_election.cc:302] T 00000000000000000000000000000000 P f2789df0324d453bbe12ff12d1cfee40 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f2789df0324d453bbe12ff12d1cfee40; no voters:
I20260620 09:30:31.618640 6265 leader_election.cc:288] T 00000000000000000000000000000000 P f2789df0324d453bbe12ff12d1cfee40 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:30:31.618690 6275 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f2789df0324d453bbe12ff12d1cfee40 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:30:31.618805 6275 raft_consensus.cc:686] T 00000000000000000000000000000000 P f2789df0324d453bbe12ff12d1cfee40 [term 1 LEADER]: Becoming Leader. State: Replica: f2789df0324d453bbe12ff12d1cfee40, State: Running, Role: LEADER
I20260620 09:30:31.618920 6265 sys_catalog.cc:564] T 00000000000000000000000000000000 P f2789df0324d453bbe12ff12d1cfee40 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:30:31.618927 6275 consensus_queue.cc:237] T 00000000000000000000000000000000 P f2789df0324d453bbe12ff12d1cfee40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2789df0324d453bbe12ff12d1cfee40" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 45763 } }
I20260620 09:30:31.619360 6276 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2789df0324d453bbe12ff12d1cfee40 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f2789df0324d453bbe12ff12d1cfee40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2789df0324d453bbe12ff12d1cfee40" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 45763 } } }
I20260620 09:30:31.619393 6277 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2789df0324d453bbe12ff12d1cfee40 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f2789df0324d453bbe12ff12d1cfee40. Latest consensus state: current_term: 1 leader_uuid: "f2789df0324d453bbe12ff12d1cfee40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2789df0324d453bbe12ff12d1cfee40" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 45763 } } }
I20260620 09:30:31.619477 6277 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2789df0324d453bbe12ff12d1cfee40 [sys.catalog]: This master's current role is: LEADER
I20260620 09:30:31.619473 6276 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2789df0324d453bbe12ff12d1cfee40 [sys.catalog]: This master's current role is: LEADER
I20260620 09:30:31.619781 6283 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:30:31.620317 6283 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:30:31.622095 6283 catalog_manager.cc:1347] Generated new cluster ID: 9dcf046843cd4ed09c1f89112c337051
I20260620 09:30:31.622164 6283 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:30:31.654840 6283 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:30:31.655050 6283 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:30:31.658725 6283 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f2789df0324d453bbe12ff12d1cfee40: Generated new TSK 0
I20260620 09:30:31.658949 6283 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 09:30:31.665160 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 6197
2026-06-20T09:30:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1338 ms)
[----------] 18 tests from AuthTokenIssuingTest (24159 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781947562949448-16558-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:30:31 dist-test-slave-706q krb5kdc[6296](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:30:31 dist-test-slave-706q krb5kdc[6296](info): set up 2 sockets
Jun 20 09:30:31 dist-test-slave-706q krb5kdc[6296](info): commencing operation
krb5kdc: starting...
W20260620 09:30:32.724872 16558 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781947562949448-16558-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781947562949448-16558-0/krb5kdc/test-user.keytab.
Jun 20 09:30:32 dist-test-slave-706q krb5kdc[6296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947832, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:30:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:30:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.43.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.43.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.43.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.190.keytab.
Entry for principal kudu/127.16.43.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.190.keytab.
I20260620 09:30:32.851495 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781947562949448-16558-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781947562949448-16558-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781947562949448-16558-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.43.190:37413
--webserver_interface=127.16.43.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.190.keytab
--principal=kudu/127.16.43.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.43.148:46023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.43.190:37413,127.16.43.189:37219,127.16.43.188:42959
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781947562949448-16558-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781947562949448-16558-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781947562949448-16558-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:30:32.932917 6309 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 09:30:32.933205 6309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:30:32.933236 6309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:30:32.933284 6309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:30:32.934484 6309 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:30:32.934564 6309 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:30:32.934589 6309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:30:32.934618 6309 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:30:32.934649 6309 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:30:32.936218 6309 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:46023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781947562949448-16558-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781947562949448-16558-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.43.190:37413,127.16.43.189:37219,127.16.43.188:42959
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.190.keytab
--principal=kudu/127.16.43.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.43.190:37413
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.43.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6309
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781947562949448-16558-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:30:32.936512 6309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:30:32.936756 6309 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:30:32.941918 6317 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:30:32.942109 6309 server_base.cc:1034] running on GCE node
W20260620 09:30:32.941975 6315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:30:32.941927 6314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:30:32.942479 6309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:30:32.942782 6309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:30:32.943979 6309 hybrid_clock.cc:648] HybridClock initialized: now 1781947832943962 us; error 46 us; skew 500 ppm
Jun 20 09:30:32 dist-test-slave-706q krb5kdc[6296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947832, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:30:32.946651 6309 init.cc:377] Logged in from keytab as kudu/127.16.43.190@KRBTEST.COM (short username kudu)
I20260620 09:30:32.947125 6309 webserver.cc:460] Webserver started at http://127.16.43.190:41735/ using document root <none> and password file <none>
I20260620 09:30:32.947364 6309 fs_manager.cc:359] Metadata directory not provided
I20260620 09:30:32.947435 6309 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:30:32.947629 6309 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:30:32.948748 6309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781947562949448-16558-0/minicluster-data/master-0/data/instance:
uuid: "33800f9b25234b5f90881a0b27c6c866"
format_stamp: "Formatted at 2026-06-20 09:30:32 on dist-test-slave-706q"
server_key: "97a5a99e6cb7911991946806e133fa22"
server_key_iv: "302858e3bad54c4e86ab13a2417ecd07"
server_key_version: "encryptionkey@0"
I20260620 09:30:32.949124 6309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781947562949448-16558-0/minicluster-data/master-0/wal/instance:
uuid: "33800f9b25234b5f90881a0b27c6c866"
format_stamp: "Formatted at 2026-06-20 09:30:32 on dist-test-slave-706q"
server_key: "97a5a99e6cb7911991946806e133fa22"
server_key_iv: "302858e3bad54c4e86ab13a2417ecd07"
server_key_version: "encryptionkey@0"
I20260620 09:30:32.950495 6309 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 09:30:32.951285 6324 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:30:32.951624 6309 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:30:32.951733 6309 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781947562949448-16558-0/minicluster-data/master-0/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781947562949448-16558-0/minicluster-data/master-0/wal
uuid: "33800f9b25234b5f90881a0b27c6c866"
format_stamp: "Formatted at 2026-06-20 09:30:32 on dist-test-slave-706q"
server_key: "97a5a99e6cb7911991946806e133fa22"
server_key_iv: "302858e3bad54c4e86ab13a2417ecd07"
server_key_version: "encryptionkey@0"
I20260620 09:30:32.951828 6309 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781947562949448-16558-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781947562949448-16558-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781947562949448-16558-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:30:32.976250 6309 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:30:32.978636 6309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:30:32.978755 6309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:30:32.982777 6309 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.190:37413
I20260620 09:30:32.982828 6376 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.190:37413 every 8 connection(s)
I20260620 09:30:32.983251 6309 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
I20260620 09:30:32.984333 6377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:30:32.985955 6377 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.43.190" port: 37413 } has no permanent_uuid. Determining permanent_uuid...
I20260620 09:30:32.986817 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 6309
I20260620 09:30:32.986948 16558 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781947562949448-16558-0/minicluster-data/master-0/wal/instance
I20260620 09:30:32.987161 16558 external_mini_cluster.cc:1428] Setting key bd8f83b4469dbb33bbbe422ccb19d008
Jun 20 09:30:32 dist-test-slave-706q krb5kdc[6296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947832, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.190@KRBTEST.COM for kudu/127.16.43.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.43.189@KRBTEST.COM; defaulting to no policy
I20260620 09:30:32.996475 6380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:32.986989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36312 (local address 127.16.43.190:37413)
0620 09:30:32.987134 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:32.987138 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:32.987171 (+ 33us) server_negotiation.cc:407] Connection header received
0620 09:30:32.987858 (+ 687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:32.987860 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:32.987885 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:32.987992 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:30:32.988560 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:32.989369 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:32.990282 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:32.990662 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:32.990796 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:32.993064 (+ 2268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:30:32.993071 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:30:32.993072 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:30:32.993079 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:30:32.994783 (+ 1704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:30:32.995201 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:30:32.995204 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:30:32.995204 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:30:32.995284 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:30:32.995517 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:30:32.995518 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:30:32.995519 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:30:32.995769 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:30:32.995867 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:32.996078 (+ 211us) server_negotiation.cc:299] Negotiation successful
0620 09:30:32.996216 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":62,"threads_started":1}
I20260620 09:30:32.996474 6379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:32.986746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:37413 (local address 127.0.0.1:36312)
0620 09:30:32.987021 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0620 09:30:32.987071 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 09:30:32.987700 (+ 629us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:30:32.988008 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:30:32.988010 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:30:32.988303 (+ 293us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:30:32.988454 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:30:32.988458 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:30:32.989490 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:30:32.989491 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:30:32.990175 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:30:32.990177 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:30:32.990783 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:30:32.990785 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:30:32.990848 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:32.991469 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:30:32.991477 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:30:32.992932 (+ 1455us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:30:32.994901 (+ 1969us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:30:32.994906 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:30:32.994908 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:30:32.995102 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:30:32.995357 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:30:32.995358 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:30:32.995359 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:30:32.995444 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:30:32.995845 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:30:32.995850 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:30:32.995912 (+ 62us) client_negotiation.cc:769] Sending connection context
0620 09:30:32.996038 (+ 126us) client_negotiation.cc:240] Negotiation successful
0620 09:30:32.996277 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"mutex_wait_us":94,"thread_start_us":101,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.43.189@KRBTEST.COM" created.
I20260620 09:30:32.997520 6377 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.43.189" port: 37219 } has no permanent_uuid. Determining permanent_uuid...
I20260620 09:30:32.997947 6379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:32.997799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.189:37219 (local address 127.0.0.1:41310)
0620 09:30:32.997870 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0620 09:30:32.997914 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.189:37219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260620 09:30:32.998124 6328 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.43.189:37219: connect: Connection refused (error 111)
I20260620 09:30:32.998613 6379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:32.998522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.189:37219 (local address 127.0.0.1:41322)
0620 09:30:32.998570 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0620 09:30:32.998596 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.189:37219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260620 09:30:32.998739 6377 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.43.189:37219: Network error: Client connection negotiation failed: client connection to 127.16.43.189:37219: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.43.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.189.keytab.
Entry for principal kudu/127.16.43.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.189.keytab.
I20260620 09:30:33.015460 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781947562949448-16558-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781947562949448-16558-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781947562949448-16558-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781947562949448-16558-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.43.189:37219
--webserver_interface=127.16.43.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.189.keytab
--principal=kudu/127.16.43.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.43.148:46023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.43.190:37413,127.16.43.189:37219,127.16.43.188:42959
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781947562949448-16558-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781947562949448-16558-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781947562949448-16558-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 09:30:33.047984 6377 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.43.189" port: 37219 } attempt: 1
I20260620 09:30:33.048629 6379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:33.048414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.189:37219 (local address 127.0.0.1:41332)
0620 09:30:33.048545 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0620 09:30:33.048593 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.189:37219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260620 09:30:33.049119 6379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:33.048997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.189:37219 (local address 127.0.0.1:41342)
0620 09:30:33.049067 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0620 09:30:33.049098 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.189:37219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260620 09:30:33.049225 6377 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.43.189:37219: Network error: Client connection negotiation failed: client connection to 127.16.43.189:37219: connect: Connection refused (error 111)
W20260620 09:30:33.097188 6384 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 09:30:33.097396 6384 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:30:33.097414 6384 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:30:33.097433 6384 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:30:33.098610 6384 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:30:33.098654 6384 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:30:33.098667 6384 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:30:33.098680 6384 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:30:33.098691 6384 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:30:33.100229 6384 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:46023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781947562949448-16558-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781947562949448-16558-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.16.43.190:37413,127.16.43.189:37219,127.16.43.188:42959
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.189.keytab
--principal=kudu/127.16.43.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.43.189:37219
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781947562949448-16558-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.16.43.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6384
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781947562949448-16558-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:30:33.100484 6384 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:30:33.100718 6384 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:30:33.105806 6390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:30:33.105821 6392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:30:33.105825 6389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:30:33.106160 6384 server_base.cc:1034] running on GCE node
I20260620 09:30:33.106413 6384 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:30:33.106700 6384 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:30:33.107863 6384 hybrid_clock.cc:648] HybridClock initialized: now 1781947833107843 us; error 37 us; skew 500 ppm
Jun 20 09:30:33 dist-test-slave-706q krb5kdc[6296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947833, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:30:33.110347 6384 init.cc:377] Logged in from keytab as kudu/127.16.43.189@KRBTEST.COM (short username kudu)
I20260620 09:30:33.110795 6384 webserver.cc:460] Webserver started at http://127.16.43.189:44305/ using document root <none> and password file <none>
I20260620 09:30:33.111055 6384 fs_manager.cc:359] Metadata directory not provided
I20260620 09:30:33.111132 6384 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:30:33.111277 6384 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:30:33.112433 6384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781947562949448-16558-0/minicluster-data/master-1/data/instance:
uuid: "bf7dd9dca41e4440b8f1c6e7809d57ef"
format_stamp: "Formatted at 2026-06-20 09:30:33 on dist-test-slave-706q"
server_key: "632833e9209306154f243c96ee5c03b3"
server_key_iv: "6112f45135f5e382568accd6df7b7f90"
server_key_version: "encryptionkey@0"
I20260620 09:30:33.112799 6384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781947562949448-16558-0/minicluster-data/master-1/wal/instance:
uuid: "bf7dd9dca41e4440b8f1c6e7809d57ef"
format_stamp: "Formatted at 2026-06-20 09:30:33 on dist-test-slave-706q"
server_key: "632833e9209306154f243c96ee5c03b3"
server_key_iv: "6112f45135f5e382568accd6df7b7f90"
server_key_version: "encryptionkey@0"
I20260620 09:30:33.114151 6384 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 09:30:33.114934 6399 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:30:33.115149 6384 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 09:30:33.115267 6384 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781947562949448-16558-0/minicluster-data/master-1/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781947562949448-16558-0/minicluster-data/master-1/wal
uuid: "bf7dd9dca41e4440b8f1c6e7809d57ef"
format_stamp: "Formatted at 2026-06-20 09:30:33 on dist-test-slave-706q"
server_key: "632833e9209306154f243c96ee5c03b3"
server_key_iv: "6112f45135f5e382568accd6df7b7f90"
server_key_version: "encryptionkey@0"
I20260620 09:30:33.115352 6384 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781947562949448-16558-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781947562949448-16558-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781947562949448-16558-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:30:33.117359 6377 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.43.189" port: 37219 } attempt: 2
I20260620 09:30:33.117769 6379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:33.117648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.189:37219 (local address 127.0.0.1:41352)
0620 09:30:33.117707 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0620 09:30:33.117742 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.189:37219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260620 09:30:33.118083 6379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:33.117992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.189:37219 (local address 127.0.0.1:41358)
0620 09:30:33.118036 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0620 09:30:33.118060 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.189:37219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260620 09:30:33.118173 6377 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.43.189:37219: Network error: Client connection negotiation failed: client connection to 127.16.43.189:37219: connect: Connection refused (error 111)
I20260620 09:30:33.129978 6384 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:30:33.132201 6384 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:30:33.132366 6384 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:30:33.136368 6384 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.189:37219
I20260620 09:30:33.136551 6451 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.189:37219 every 8 connection(s)
I20260620 09:30:33.136809 6384 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781947562949448-16558-0/minicluster-data/master-1/data/info.pb
I20260620 09:30:33.137715 6452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:30:33.139336 6452 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.43.190" port: 37413 } has no permanent_uuid. Determining permanent_uuid...
I20260620 09:30:33.140453 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 6384
I20260620 09:30:33.140618 16558 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781947562949448-16558-0/minicluster-data/master-1/wal/instance
I20260620 09:30:33.140801 16558 external_mini_cluster.cc:1428] Setting key 490219c30ab92c3f650e16bcc4762999
Jun 20 09:30:33 dist-test-slave-706q krb5kdc[6296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947833, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.189@KRBTEST.COM for kudu/127.16.43.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.43.188@KRBTEST.COM; defaulting to no policy
I20260620 09:30:33.149549 6380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:33.140132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36318 (local address 127.16.43.190:37413)
0620 09:30:33.140193 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:33.140195 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:33.140391 (+ 196us) server_negotiation.cc:407] Connection header received
0620 09:30:33.141027 (+ 636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:33.141028 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:33.141045 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:33.141116 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:30:33.141753 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:33.142590 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:33.143658 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:33.143910 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:33.143966 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:33.146466 (+ 2500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:30:33.146468 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:30:33.146468 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:30:33.146477 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:30:33.148015 (+ 1538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:30:33.148489 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:30:33.148490 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:30:33.148491 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:30:33.148556 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:30:33.148829 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:30:33.148830 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:30:33.148830 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:30:33.148970 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:30:33.149047 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:33.149359 (+ 312us) server_negotiation.cc:299] Negotiation successful
0620 09:30:33.149451 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 09:30:33.149760 6454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:33.140066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:37413 (local address 127.0.0.1:36318)
0620 09:30:33.140249 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0620 09:30:33.140290 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 09:30:33.140915 (+ 625us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:30:33.141189 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:30:33.141193 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:30:33.141450 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:30:33.141650 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:30:33.141654 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:30:33.142695 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:30:33.142697 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:30:33.143505 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:30:33.143508 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:30:33.144009 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:30:33.144010 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:30:33.144192 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:33.144748 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:30:33.144759 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:30:33.146360 (+ 1601us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:30:33.148149 (+ 1789us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:30:33.148166 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:30:33.148168 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:30:33.148405 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:30:33.148639 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:30:33.148640 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:30:33.148641 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:30:33.148754 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:30:33.149051 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:30:33.149059 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:30:33.149224 (+ 165us) client_negotiation.cc:769] Sending connection context
0620 09:30:33.149338 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 09:30:33.149507 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":106,"threads_started":1}
I20260620 09:30:33.150482 6452 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.43.189" port: 37219 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.43.188@KRBTEST.COM" created.
Jun 20 09:30:33 dist-test-slave-706q krb5kdc[6296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947833, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.189@KRBTEST.COM for kudu/127.16.43.189@KRBTEST.COM
I20260620 09:30:33.159161 6454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:33.150832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.189:37219 (local address 127.0.0.1:41362)
0620 09:30:33.150895 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0620 09:30:33.150906 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 09:30:33.150973 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:30:33.151244 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:30:33.151245 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:30:33.151386 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:30:33.151444 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:30:33.151446 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:30:33.152690 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:30:33.152691 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:30:33.153196 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:30:33.153198 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:30:33.153822 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:30:33.153823 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:30:33.153913 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:33.154563 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:30:33.154569 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:30:33.156133 (+ 1564us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:30:33.158055 (+ 1922us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:30:33.158056 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:30:33.158057 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:30:33.158218 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:30:33.158455 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:30:33.158455 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:30:33.158456 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:30:33.158524 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:30:33.158821 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:30:33.158822 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:30:33.158861 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 09:30:33.158933 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 09:30:33.159041 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260620 09:30:33.159224 6456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:33.150915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41362 (local address 127.16.43.189:37219)
0620 09:30:33.151072 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:33.151076 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:33.151095 (+ 19us) server_negotiation.cc:407] Connection header received
0620 09:30:33.151118 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:33.151120 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:33.151137 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:33.151203 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:30:33.151574 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:33.152434 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:33.153283 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:33.153728 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:33.153787 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:33.156240 (+ 2453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:30:33.156249 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:30:33.156250 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:30:33.156256 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:30:33.157926 (+ 1670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:30:33.158324 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:30:33.158327 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:30:33.158327 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:30:33.158383 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:30:33.158588 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:30:33.158589 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:30:33.158589 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:30:33.158748 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:30:33.158844 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:33.158965 (+ 121us) server_negotiation.cc:299] Negotiation successful
0620 09:30:33.159095 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":128,"thread_start_us":100,"threads_started":1}
I20260620 09:30:33.159873 6452 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.43.188" port: 42959 } has no permanent_uuid. Determining permanent_uuid...
I20260620 09:30:33.160295 6454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:33.160168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.188:42959 (local address 127.0.0.1:43394)
0620 09:30:33.160220 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0620 09:30:33.160269 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.188:42959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260620 09:30:33.160434 6401 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.43.188:42959: connect: Connection refused (error 111)
I20260620 09:30:33.160841 6454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:33.160746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.188:42959 (local address 127.0.0.1:43402)
0620 09:30:33.160797 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0620 09:30:33.160821 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.188:42959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260620 09:30:33.160948 6452 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.43.188:42959: Network error: Client connection negotiation failed: client connection to 127.16.43.188:42959: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.43.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.188.keytab.
Entry for principal kudu/127.16.43.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.188.keytab.
I20260620 09:30:33.171098 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781947562949448-16558-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781947562949448-16558-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781947562949448-16558-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781947562949448-16558-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.43.188:42959
--webserver_interface=127.16.43.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.188.keytab
--principal=kudu/127.16.43.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.43.148:46023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.43.190:37413,127.16.43.189:37219,127.16.43.188:42959
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781947562949448-16558-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781947562949448-16558-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781947562949448-16558-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 09:30:33.209338 6377 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.43.189" port: 37219 } attempt: 3
I20260620 09:30:33.210147 6452 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.43.188" port: 42959 } attempt: 1
I20260620 09:30:33.210566 6454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:33.210417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.188:42959 (local address 127.0.0.1:43412)
0620 09:30:33.210492 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0620 09:30:33.210532 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.188:42959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260620 09:30:33.210898 6454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:33.210803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.188:42959 (local address 127.0.0.1:43414)
0620 09:30:33.210853 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0620 09:30:33.210880 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.188:42959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260620 09:30:33.210979 6452 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.43.188:42959: Network error: Client connection negotiation failed: client connection to 127.16.43.188:42959: connect: Connection refused (error 111)
Jun 20 09:30:33 dist-test-slave-706q krb5kdc[6296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947832, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.190@KRBTEST.COM for kudu/127.16.43.189@KRBTEST.COM
I20260620 09:30:33.217229 6379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:33.209738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.189:37219 (local address 127.0.0.1:41368)
0620 09:30:33.209802 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0620 09:30:33.209816 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 09:30:33.209870 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:30:33.210117 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:30:33.210118 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:30:33.210320 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:30:33.210404 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:30:33.210407 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:30:33.211392 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:30:33.211393 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:30:33.212005 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:30:33.212007 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:30:33.212438 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:30:33.212439 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:30:33.212533 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:33.213051 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:30:33.213056 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:30:33.214334 (+ 1278us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:30:33.216212 (+ 1878us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:30:33.216212 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:30:33.216213 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:30:33.216381 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:30:33.216595 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:30:33.216595 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:30:33.216596 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:30:33.216652 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:30:33.216931 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:30:33.216932 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:30:33.217003 (+ 71us) client_negotiation.cc:769] Sending connection context
0620 09:30:33.217076 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 09:30:33.217160 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260620 09:30:33.217308 6456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:33.209837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41368 (local address 127.16.43.189:37219)
0620 09:30:33.209914 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:33.209917 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:33.209927 (+ 10us) server_negotiation.cc:407] Connection header received
0620 09:30:33.209964 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:33.209966 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:33.209977 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:33.210055 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:30:33.210495 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:33.211299 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:33.212114 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:33.212347 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:33.212437 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:33.214437 (+ 2000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:30:33.214438 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:30:33.214438 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:30:33.214446 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:30:33.216119 (+ 1673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:30:33.216463 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:30:33.216464 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:30:33.216464 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:30:33.216527 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:30:33.216713 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:30:33.216714 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:30:33.216714 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:30:33.216848 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:30:33.216914 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:33.217130 (+ 216us) server_negotiation.cc:299] Negotiation successful
0620 09:30:33.217221 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 09:30:33.217772 6377 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.43.188" port: 42959 } has no permanent_uuid. Determining permanent_uuid...
I20260620 09:30:33.218254 6379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:33.218117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.188:42959 (local address 127.0.0.1:43430)
0620 09:30:33.218188 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0620 09:30:33.218229 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.188:42959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260620 09:30:33.218614 6379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:33.218495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.188:42959 (local address 127.0.0.1:43440)
0620 09:30:33.218554 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0620 09:30:33.218592 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.188:42959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260620 09:30:33.218740 6377 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.43.188:42959: Network error: Client connection negotiation failed: client connection to 127.16.43.188:42959: connect: Connection refused (error 111)
I20260620 09:30:33.249922 6377 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.43.188" port: 42959 } attempt: 1
I20260620 09:30:33.250487 6379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:33.250329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.188:42959 (local address 127.0.0.1:43450)
0620 09:30:33.250409 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0620 09:30:33.250457 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.188:42959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260620 09:30:33.250967 6379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:33.250841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.188:42959 (local address 127.0.0.1:43462)
0620 09:30:33.250912 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0620 09:30:33.250943 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.188:42959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260620 09:30:33.251117 6377 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.43.188:42959: Network error: Client connection negotiation failed: client connection to 127.16.43.188:42959: connect: Connection refused (error 111)
W20260620 09:30:33.255919 6459 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 09:30:33.256132 6459 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:30:33.256150 6459 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:30:33.256172 6459 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:30:33.257305 6459 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:30:33.257342 6459 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:30:33.257356 6459 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:30:33.257368 6459 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:30:33.257380 6459 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:30:33.258795 6459 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:46023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781947562949448-16558-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781947562949448-16558-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.16.43.190:37413,127.16.43.189:37219,127.16.43.188:42959
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.188.keytab
--principal=kudu/127.16.43.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.43.188:42959
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781947562949448-16558-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.16.43.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6459
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781947562949448-16558-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:30:33.258997 6459 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:30:33.259195 6459 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:30:33.264354 6465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:30:33.264413 6464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:30:33.264520 6459 server_base.cc:1034] running on GCE node
W20260620 09:30:33.264354 6467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:30:33.264950 6459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:30:33.265229 6459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:30:33.266399 6459 hybrid_clock.cc:648] HybridClock initialized: now 1781947833266390 us; error 31 us; skew 500 ppm
Jun 20 09:30:33 dist-test-slave-706q krb5kdc[6296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947833, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:30:33.269079 6459 init.cc:377] Logged in from keytab as kudu/127.16.43.188@KRBTEST.COM (short username kudu)
I20260620 09:30:33.269583 6459 webserver.cc:460] Webserver started at http://127.16.43.188:39175/ using document root <none> and password file <none>
I20260620 09:30:33.269819 6459 fs_manager.cc:359] Metadata directory not provided
I20260620 09:30:33.269891 6459 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:30:33.270047 6459 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:30:33.271117 6459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781947562949448-16558-0/minicluster-data/master-2/data/instance:
uuid: "25cf6661c27944438315b0dacb2037a1"
format_stamp: "Formatted at 2026-06-20 09:30:33 on dist-test-slave-706q"
server_key: "744691973502f9377b5a0164ba684286"
server_key_iv: "e98382fa1079d51084fd1871d8bc9b10"
server_key_version: "encryptionkey@0"
I20260620 09:30:33.271497 6459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781947562949448-16558-0/minicluster-data/master-2/wal/instance:
uuid: "25cf6661c27944438315b0dacb2037a1"
format_stamp: "Formatted at 2026-06-20 09:30:33 on dist-test-slave-706q"
server_key: "744691973502f9377b5a0164ba684286"
server_key_iv: "e98382fa1079d51084fd1871d8bc9b10"
server_key_version: "encryptionkey@0"
I20260620 09:30:33.272864 6459 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:30:33.273731 6474 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:30:33.273913 6459 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:30:33.274016 6459 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781947562949448-16558-0/minicluster-data/master-2/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781947562949448-16558-0/minicluster-data/master-2/wal
uuid: "25cf6661c27944438315b0dacb2037a1"
format_stamp: "Formatted at 2026-06-20 09:30:33 on dist-test-slave-706q"
server_key: "744691973502f9377b5a0164ba684286"
server_key_iv: "e98382fa1079d51084fd1871d8bc9b10"
server_key_version: "encryptionkey@0"
I20260620 09:30:33.274106 6459 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781947562949448-16558-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781947562949448-16558-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781947562949448-16558-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:30:33.279102 6452 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.43.188" port: 42959 } attempt: 2
I20260620 09:30:33.279518 6454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:33.279387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.188:42959 (local address 127.0.0.1:43472)
0620 09:30:33.279447 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0620 09:30:33.279489 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.188:42959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260620 09:30:33.279933 6454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:33.279840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.188:42959 (local address 127.0.0.1:43476)
0620 09:30:33.279884 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0620 09:30:33.279909 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.188:42959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260620 09:30:33.280021 6452 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.43.188:42959: Network error: Client connection negotiation failed: client connection to 127.16.43.188:42959: connect: Connection refused (error 111)
I20260620 09:30:33.286586 6459 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:30:33.288813 6459 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:30:33.288988 6459 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:30:33.293221 6459 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.188:42959
I20260620 09:30:33.293255 6526 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.188:42959 every 8 connection(s)
I20260620 09:30:33.293630 6459 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781947562949448-16558-0/minicluster-data/master-2/data/info.pb
I20260620 09:30:33.294689 6527 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:30:33.296367 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 6459
I20260620 09:30:33.296504 16558 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781947562949448-16558-0/minicluster-data/master-2/wal/instance
I20260620 09:30:33.296346 6527 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.43.190" port: 37413 } has no permanent_uuid. Determining permanent_uuid...
I20260620 09:30:33.296677 16558 external_mini_cluster.cc:1428] Setting key 5e6cbbbd1f28d31d51702b4e904268ac
Jun 20 09:30:33 dist-test-slave-706q krb5kdc[6296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947833, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.188@KRBTEST.COM for kudu/127.16.43.190@KRBTEST.COM
Jun 20 09:30:33 dist-test-slave-706q krb5kdc[6296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947832, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.43.188@KRBTEST.COM
I20260620 09:30:33.307273 6380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:33.297264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36330 (local address 127.16.43.190:37413)
0620 09:30:33.297333 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:33.297336 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:33.297566 (+ 230us) server_negotiation.cc:407] Connection header received
0620 09:30:33.298184 (+ 618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:33.298185 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:33.298198 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:33.298262 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:30:33.298807 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:33.299287 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:33.300413 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:33.300707 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:33.300790 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:33.303289 (+ 2499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:30:33.303291 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:30:33.303291 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:30:33.303298 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:30:33.305589 (+ 2291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:30:33.306239 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:30:33.306240 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:30:33.306241 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:30:33.306307 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:30:33.306595 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:30:33.306596 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:30:33.306596 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:30:33.306744 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:30:33.306816 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:33.307077 (+ 261us) server_negotiation.cc:299] Negotiation successful
0620 09:30:33.307163 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 09:30:33.307482 6531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:33.297411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43484 (local address 127.16.43.188:42959)
0620 09:30:33.297526 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:33.297530 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:33.297647 (+ 117us) server_negotiation.cc:407] Connection header received
0620 09:30:33.298053 (+ 406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:33.298062 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:33.298089 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:33.298178 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:30:33.298859 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:33.299920 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:33.300947 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:33.301355 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:33.301440 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:33.303956 (+ 2516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:30:33.303964 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:30:33.303964 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:30:33.303972 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:30:33.305841 (+ 1869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:30:33.306258 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:30:33.306261 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:30:33.306261 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:30:33.306372 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:30:33.306609 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:30:33.306610 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:30:33.306610 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:30:33.306909 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:30:33.307055 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:33.307166 (+ 111us) server_negotiation.cc:299] Negotiation successful
0620 09:30:33.307279 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":38,"server-negotiator.queue_time_us":89,"thread_start_us":45,"threads_started":1}
I20260620 09:30:33.307909 6530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:33.297240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:37413 (local address 127.0.0.1:36330)
0620 09:30:33.297392 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 09:30:33.297438 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 09:30:33.298051 (+ 613us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:30:33.298303 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:30:33.298305 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:30:33.298560 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:30:33.298713 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:30:33.298717 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:30:33.299368 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:30:33.299369 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:30:33.300326 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:30:33.300329 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:30:33.300813 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:30:33.300814 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:30:33.300929 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:33.301519 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:30:33.301533 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:30:33.303188 (+ 1655us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:30:33.305706 (+ 2518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:30:33.305713 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:30:33.305715 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:30:33.306133 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:30:33.306381 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:30:33.306382 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:30:33.306382 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:30:33.306521 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:30:33.306833 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:30:33.306840 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:30:33.306900 (+ 60us) client_negotiation.cc:769] Sending connection context
0620 09:30:33.307040 (+ 140us) client_negotiation.cc:240] Negotiation successful
0620 09:30:33.307227 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"mutex_wait_us":33,"thread_start_us":73,"threads_started":1}
I20260620 09:30:33.308576 6527 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.43.189" port: 37219 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 09:30:33 dist-test-slave-706q krb5kdc[6296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947833, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.188@KRBTEST.COM for kudu/127.16.43.189@KRBTEST.COM
Jun 20 09:30:33 dist-test-slave-706q krb5kdc[6296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947832, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.43.190@KRBTEST.COM
I20260620 09:30:33.316145 6530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:33.308852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.189:37219 (local address 127.0.0.1:41372)
0620 09:30:33.308908 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0620 09:30:33.308919 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 09:30:33.308973 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:30:33.309092 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:30:33.309093 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:30:33.309238 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:30:33.309273 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:30:33.309274 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:30:33.310106 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:30:33.310107 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:30:33.310854 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:30:33.310857 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:30:33.311335 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:30:33.311336 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:30:33.311398 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:33.311801 (+ 403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:30:33.311805 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:30:33.313141 (+ 1336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:30:33.314913 (+ 1772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:30:33.314914 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:30:33.314914 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:30:33.315163 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:30:33.315375 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:30:33.315376 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:30:33.315377 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:30:33.315555 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:30:33.315814 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:30:33.315815 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:30:33.315861 (+ 46us) client_negotiation.cc:769] Sending connection context
0620 09:30:33.315947 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 09:30:33.316043 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260620 09:30:33.316143 6456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:33.308908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41372 (local address 127.16.43.189:37219)
0620 09:30:33.308950 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:33.308952 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:33.308960 (+ 8us) server_negotiation.cc:407] Connection header received
0620 09:30:33.309031 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:33.309031 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:33.309039 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:33.309078 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:30:33.309350 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:33.310027 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:33.310938 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:33.311263 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:33.311321 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:33.313233 (+ 1912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:30:33.313235 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:30:33.313235 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:30:33.313242 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:30:33.314792 (+ 1550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:30:33.315252 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:30:33.315253 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:30:33.315254 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:30:33.315305 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:30:33.315618 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:30:33.315619 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:30:33.315619 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:30:33.315733 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:30:33.315822 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:33.315974 (+ 152us) server_negotiation.cc:299] Negotiation successful
0620 09:30:33.316043 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 09:30:33.316620 6527 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.43.188" port: 42959 } has no permanent_uuid. Determining permanent_uuid...
I20260620 09:30:33.317204 6380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:33.308412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36340 (local address 127.16.43.190:37413)
0620 09:30:33.308450 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:33.308452 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:33.308461 (+ 9us) server_negotiation.cc:407] Connection header received
0620 09:30:33.308518 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:33.308519 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:33.308541 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:33.308595 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:30:33.309171 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:33.309859 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:33.310938 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:33.311230 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:33.311299 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:33.313843 (+ 2544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:30:33.313844 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:30:33.313845 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:30:33.313852 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:30:33.315687 (+ 1835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:30:33.316031 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:30:33.316033 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:30:33.316033 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:30:33.316097 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:30:33.316318 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:30:33.316319 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:30:33.316320 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:30:33.316463 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:30:33.316626 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:33.317019 (+ 393us) server_negotiation.cc:299] Negotiation successful
0620 09:30:33.317092 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
Jun 20 09:30:33 dist-test-slave-706q krb5kdc[6296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947833, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.188@KRBTEST.COM for kudu/127.16.43.188@KRBTEST.COM
Jun 20 09:30:33 dist-test-slave-706q krb5kdc[6296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947832, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.43.189@KRBTEST.COM
I20260620 09:30:33.323484 6530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:33.316880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.188:42959 (local address 127.0.0.1:43500)
0620 09:30:33.316923 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0620 09:30:33.316932 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 09:30:33.316966 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:30:33.317123 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:30:33.317124 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:30:33.317264 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:30:33.317300 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:30:33.317302 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:30:33.318115 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:30:33.318116 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:30:33.318592 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:30:33.318593 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:30:33.319014 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:30:33.319015 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:30:33.319063 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:33.319491 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:30:33.319495 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:30:33.320679 (+ 1184us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:30:33.322410 (+ 1731us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:30:33.322410 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:30:33.322411 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:30:33.322624 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:30:33.322884 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:30:33.322884 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:30:33.322885 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:30:33.322937 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:30:33.323234 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:30:33.323235 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:30:33.323283 (+ 48us) client_negotiation.cc:769] Sending connection context
0620 09:30:33.323351 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 09:30:33.323421 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260620 09:30:33.323647 6531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:33.316965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43500 (local address 127.16.43.188:42959)
0620 09:30:33.317012 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:33.317013 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:33.317021 (+ 8us) server_negotiation.cc:407] Connection header received
0620 09:30:33.317041 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:33.317042 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:33.317050 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:33.317100 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:30:33.317378 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:33.318004 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:33.318693 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:33.318913 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:33.318994 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:33.320774 (+ 1780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:30:33.320775 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:30:33.320775 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:30:33.320782 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:30:33.322328 (+ 1546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:30:33.322712 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:30:33.322713 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:30:33.322713 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:30:33.322803 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:30:33.323007 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:30:33.323008 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:30:33.323009 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:30:33.323134 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:30:33.323222 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:33.323401 (+ 179us) server_negotiation.cc:299] Negotiation successful
0620 09:30:33.323480 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 09:30:33.325304 6527 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25cf6661c27944438315b0dacb2037a1: Bootstrap starting.
I20260620 09:30:33.325845 6456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:33.317891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41386 (local address 127.16.43.189:37219)
0620 09:30:33.317933 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:33.317935 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:33.317944 (+ 9us) server_negotiation.cc:407] Connection header received
0620 09:30:33.317988 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:33.317989 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:33.317998 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:33.318039 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:30:33.318542 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:33.318985 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:33.319894 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:33.320119 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:33.320207 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:33.322806 (+ 2599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:30:33.322808 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:30:33.322808 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:30:33.322815 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:30:33.324530 (+ 1715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:30:33.324941 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:30:33.324941 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:30:33.324942 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:30:33.324988 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:30:33.325220 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:30:33.325221 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:30:33.325221 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:30:33.325334 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:30:33.325598 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:33.325716 (+ 118us) server_negotiation.cc:299] Negotiation successful
0620 09:30:33.325778 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 09:30:33.326267 6377 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.43.188" port: 42959 } attempt: 2
I20260620 09:30:33.326342 6527 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 25cf6661c27944438315b0dacb2037a1: Neither blocks nor log segments found. Creating new log.
I20260620 09:30:33.326679 6527 log.cc:824] T 00000000000000000000000000000000 P 25cf6661c27944438315b0dacb2037a1: Log is configured to *not* fsync() on all Append() calls
I20260620 09:30:33.327575 6527 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25cf6661c27944438315b0dacb2037a1: No bootstrap required, opened a new log
I20260620 09:30:33.328054 16558 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 09:30:33.329571 6527 raft_consensus.cc:348] T 00000000000000000000000000000000 P 25cf6661c27944438315b0dacb2037a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33800f9b25234b5f90881a0b27c6c866" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 37413 } } peers { permanent_uuid: "bf7dd9dca41e4440b8f1c6e7809d57ef" member_type: VOTER last_known_addr { host: "127.16.43.189" port: 37219 } } peers { permanent_uuid: "25cf6661c27944438315b0dacb2037a1" member_type: VOTER last_known_addr { host: "127.16.43.188" port: 42959 } }
I20260620 09:30:33.329728 6527 raft_consensus.cc:729] T 00000000000000000000000000000000 P 25cf6661c27944438315b0dacb2037a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25cf6661c27944438315b0dacb2037a1, State: Initialized, Role: FOLLOWER
I20260620 09:30:33.329789 6527 consensus_queue.cc:260] T 00000000000000000000000000000000 P 25cf6661c27944438315b0dacb2037a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33800f9b25234b5f90881a0b27c6c866" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 37413 } } peers { permanent_uuid: "bf7dd9dca41e4440b8f1c6e7809d57ef" member_type: VOTER last_known_addr { host: "127.16.43.189" port: 37219 } } peers { permanent_uuid: "25cf6661c27944438315b0dacb2037a1" member_type: VOTER last_known_addr { host: "127.16.43.188" port: 42959 } }
I20260620 09:30:33.328935 6351 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "33800f9b25234b5f90881a0b27c6c866"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36340
I20260620 09:30:33.330119 6527 sys_catalog.cc:564] T 00000000000000000000000000000000 P 25cf6661c27944438315b0dacb2037a1 [sys.catalog]: configured and running, proceeding with master startup.
Jun 20 09:30:33 dist-test-slave-706q krb5kdc[6296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947832, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.190@KRBTEST.COM for kudu/127.16.43.188@KRBTEST.COM
W20260620 09:30:33.332052 6544 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 25cf6661c27944438315b0dacb2037a1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 09:30:33.332113 6544 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 09:30:33.331612 6533 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25cf6661c27944438315b0dacb2037a1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33800f9b25234b5f90881a0b27c6c866" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 37413 } } peers { permanent_uuid: "bf7dd9dca41e4440b8f1c6e7809d57ef" member_type: VOTER last_known_addr { host: "127.16.43.189" port: 37219 } } peers { permanent_uuid: "25cf6661c27944438315b0dacb2037a1" member_type: VOTER last_known_addr { host: "127.16.43.188" port: 42959 } } }
I20260620 09:30:33.332163 6533 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25cf6661c27944438315b0dacb2037a1 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 09:30:33.334466 6379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:33.326578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.188:42959 (local address 127.0.0.1:43516)
0620 09:30:33.326730 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 09:30:33.326740 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 09:30:33.326789 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:30:33.326962 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:30:33.326963 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:30:33.327139 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:30:33.327192 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:30:33.327195 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:30:33.328189 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:30:33.328191 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:30:33.328975 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:30:33.328977 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:30:33.329426 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:30:33.329427 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:30:33.329489 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:33.329921 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:30:33.329926 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:30:33.331408 (+ 1482us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:30:33.333276 (+ 1868us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:30:33.333277 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:30:33.333277 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:30:33.333451 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:30:33.333676 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:30:33.333677 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:30:33.333677 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:30:33.333735 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:30:33.334086 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:30:33.334087 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:30:33.334131 (+ 44us) client_negotiation.cc:769] Sending connection context
0620 09:30:33.334278 (+ 147us) client_negotiation.cc:240] Negotiation successful
0620 09:30:33.334376 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133}
I20260620 09:30:33.334463 6531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:33.326662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43516 (local address 127.16.43.188:42959)
0620 09:30:33.326723 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:33.326725 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:33.326773 (+ 48us) server_negotiation.cc:407] Connection header received
0620 09:30:33.326852 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:33.326853 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:33.326866 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:33.327330 (+ 464us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:30:33.327357 (+ 27us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:33.328097 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:33.329065 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:33.329337 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:33.329411 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:33.331485 (+ 2074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:30:33.331486 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:30:33.331487 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:30:33.331493 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:30:33.333207 (+ 1714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:30:33.333547 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:30:33.333548 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:30:33.333549 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:30:33.333604 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:30:33.333834 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:30:33.333835 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:30:33.333836 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:30:33.334016 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:30:33.334081 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:33.334307 (+ 226us) server_negotiation.cc:299] Negotiation successful
0620 09:30:33.334377 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 09:30:33.332595 6459 master_runner.cc:186] Error getting master registration for 127.16.43.190:37413: OK, instance_id { permanent_uuid: "33800f9b25234b5f90881a0b27c6c866" instance_seqno: 1781947832981608 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 09:30:33.335471 6459 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 09:30:33.335776 6377 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33800f9b25234b5f90881a0b27c6c866: Bootstrap starting.
I20260620 09:30:33.336354 6377 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 33800f9b25234b5f90881a0b27c6c866: Neither blocks nor log segments found. Creating new log.
I20260620 09:30:33.336638 6377 log.cc:824] T 00000000000000000000000000000000 P 33800f9b25234b5f90881a0b27c6c866: Log is configured to *not* fsync() on all Append() calls
I20260620 09:30:33.337337 6377 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33800f9b25234b5f90881a0b27c6c866: No bootstrap required, opened a new log
I20260620 09:30:33.338409 6377 raft_consensus.cc:348] T 00000000000000000000000000000000 P 33800f9b25234b5f90881a0b27c6c866 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33800f9b25234b5f90881a0b27c6c866" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 37413 } } peers { permanent_uuid: "bf7dd9dca41e4440b8f1c6e7809d57ef" member_type: VOTER last_known_addr { host: "127.16.43.189" port: 37219 } } peers { permanent_uuid: "25cf6661c27944438315b0dacb2037a1" member_type: VOTER last_known_addr { host: "127.16.43.188" port: 42959 } }
I20260620 09:30:33.338536 6377 raft_consensus.cc:729] T 00000000000000000000000000000000 P 33800f9b25234b5f90881a0b27c6c866 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33800f9b25234b5f90881a0b27c6c866, State: Initialized, Role: FOLLOWER
I20260620 09:30:33.338656 6377 consensus_queue.cc:260] T 00000000000000000000000000000000 P 33800f9b25234b5f90881a0b27c6c866 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33800f9b25234b5f90881a0b27c6c866" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 37413 } } peers { permanent_uuid: "bf7dd9dca41e4440b8f1c6e7809d57ef" member_type: VOTER last_known_addr { host: "127.16.43.189" port: 37219 } } peers { permanent_uuid: "25cf6661c27944438315b0dacb2037a1" member_type: VOTER last_known_addr { host: "127.16.43.188" port: 42959 } }
I20260620 09:30:33.338969 6377 sys_catalog.cc:564] T 00000000000000000000000000000000 P 33800f9b25234b5f90881a0b27c6c866 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:30:33.339011 6351 raft_consensus.cc:482] T 00000000000000000000000000000000 P 33800f9b25234b5f90881a0b27c6c866 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 09:30:33.339051 6351 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 33800f9b25234b5f90881a0b27c6c866 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:30:33.339989 6351 raft_consensus.cc:504] T 00000000000000000000000000000000 P 33800f9b25234b5f90881a0b27c6c866 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33800f9b25234b5f90881a0b27c6c866" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 37413 } } peers { permanent_uuid: "bf7dd9dca41e4440b8f1c6e7809d57ef" member_type: VOTER last_known_addr { host: "127.16.43.189" port: 37219 } } peers { permanent_uuid: "25cf6661c27944438315b0dacb2037a1" member_type: VOTER last_known_addr { host: "127.16.43.188" port: 42959 } }
I20260620 09:30:33.340268 6351 leader_election.cc:288] T 00000000000000000000000000000000 P 33800f9b25234b5f90881a0b27c6c866 [CANDIDATE]: Term 1 election: Requested vote from peers bf7dd9dca41e4440b8f1c6e7809d57ef (127.16.43.189:37219), 25cf6661c27944438315b0dacb2037a1 (127.16.43.188:42959)
I20260620 09:30:33.338949 6547 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33800f9b25234b5f90881a0b27c6c866 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33800f9b25234b5f90881a0b27c6c866" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 37413 } } peers { permanent_uuid: "bf7dd9dca41e4440b8f1c6e7809d57ef" member_type: VOTER last_known_addr { host: "127.16.43.189" port: 37219 } } peers { permanent_uuid: "25cf6661c27944438315b0dacb2037a1" member_type: VOTER last_known_addr { host: "127.16.43.188" port: 42959 } } }
I20260620 09:30:33.340837 6547 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33800f9b25234b5f90881a0b27c6c866 [sys.catalog]: This master's current role is: FOLLOWER
W20260620 09:30:33.341148 6558 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 33800f9b25234b5f90881a0b27c6c866: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 09:30:33.341219 6558 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 09:30:33.341225 6501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "33800f9b25234b5f90881a0b27c6c866" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "25cf6661c27944438315b0dacb2037a1"
I20260620 09:30:33.341337 6501 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 25cf6661c27944438315b0dacb2037a1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:30:33.342674 6501 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 25cf6661c27944438315b0dacb2037a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33800f9b25234b5f90881a0b27c6c866 in term 1.
I20260620 09:30:33.342917 6326 leader_election.cc:302] T 00000000000000000000000000000000 P 33800f9b25234b5f90881a0b27c6c866 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25cf6661c27944438315b0dacb2037a1, 33800f9b25234b5f90881a0b27c6c866; no voters:
I20260620 09:30:33.343096 6547 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 33800f9b25234b5f90881a0b27c6c866 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:30:33.343259 6547 raft_consensus.cc:686] T 00000000000000000000000000000000 P 33800f9b25234b5f90881a0b27c6c866 [term 1 LEADER]: Becoming Leader. State: Replica: 33800f9b25234b5f90881a0b27c6c866, State: Running, Role: LEADER
I20260620 09:30:33.343343 6547 consensus_queue.cc:237] T 00000000000000000000000000000000 P 33800f9b25234b5f90881a0b27c6c866 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33800f9b25234b5f90881a0b27c6c866" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 37413 } } peers { permanent_uuid: "bf7dd9dca41e4440b8f1c6e7809d57ef" member_type: VOTER last_known_addr { host: "127.16.43.189" port: 37219 } } peers { permanent_uuid: "25cf6661c27944438315b0dacb2037a1" member_type: VOTER last_known_addr { host: "127.16.43.188" port: 42959 } }
I20260620 09:30:33.343920 6547 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33800f9b25234b5f90881a0b27c6c866 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 33800f9b25234b5f90881a0b27c6c866. Latest consensus state: current_term: 1 leader_uuid: "33800f9b25234b5f90881a0b27c6c866" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33800f9b25234b5f90881a0b27c6c866" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 37413 } } peers { permanent_uuid: "bf7dd9dca41e4440b8f1c6e7809d57ef" member_type: VOTER last_known_addr { host: "127.16.43.189" port: 37219 } } peers { permanent_uuid: "25cf6661c27944438315b0dacb2037a1" member_type: VOTER last_known_addr { host: "127.16.43.188" port: 42959 } } }
I20260620 09:30:33.344048 6547 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33800f9b25234b5f90881a0b27c6c866 [sys.catalog]: This master's current role is: LEADER
I20260620 09:30:33.344267 6567 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:30:33.344507 6567 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:30:33.343458 6309 master_runner.cc:186] Error getting master registration for 127.16.43.189:37219: OK, instance_id { permanent_uuid: "bf7dd9dca41e4440b8f1c6e7809d57ef" instance_seqno: 1781947833135100 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 09:30:33.346217 6309 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 09:30:33.356346 6380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:33.346265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36348 (local address 127.16.43.190:37413)
0620 09:30:33.346306 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:33.346308 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:33.346998 (+ 690us) server_negotiation.cc:407] Connection header received
0620 09:30:33.347144 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:33.347145 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:33.347168 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:33.347201 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:30:33.348331 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:33.349021 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:33.350441 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:33.350747 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:33.350794 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:33.352459 (+ 1665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:30:33.352461 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:30:33.352462 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:30:33.352470 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:30:33.354255 (+ 1785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:30:33.354820 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:30:33.354822 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:30:33.354822 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:30:33.354884 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:30:33.355290 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:30:33.355291 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:30:33.355291 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:30:33.355454 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:30:33.355571 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:33.356122 (+ 551us) server_negotiation.cc:299] Negotiation successful
0620 09:30:33.356216 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 09:30:33.360707 6456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:33.342153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41394 (local address 127.16.43.189:37219)
0620 09:30:33.342211 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:33.342214 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:33.346631 (+ 4417us) server_negotiation.cc:407] Connection header received
0620 09:30:33.346747 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:33.346748 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:33.346762 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:33.346796 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:30:33.347695 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:33.348481 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:33.349462 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:33.349821 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:33.349867 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:33.355808 (+ 5941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:30:33.355809 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:30:33.355809 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:30:33.355820 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:30:33.358005 (+ 2185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:30:33.359778 (+ 1773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:30:33.359780 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:30:33.359780 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:30:33.359845 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:30:33.360069 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:30:33.360070 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:30:33.360071 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:30:33.360222 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:30:33.360304 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:33.360508 (+ 204us) server_negotiation.cc:299] Negotiation successful
0620 09:30:33.360599 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 09:30:33.360693 6501 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 25cf6661c27944438315b0dacb2037a1 [term 1 FOLLOWER]: Refusing update from remote peer 33800f9b25234b5f90881a0b27c6c866: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:30:33.361025 6565 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 33800f9b25234b5f90881a0b27c6c866 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25cf6661c27944438315b0dacb2037a1" member_type: VOTER last_known_addr { host: "127.16.43.188" port: 42959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:30:33.362327 6531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:33.346756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43528 (local address 127.16.43.188:42959)
0620 09:30:33.346806 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:33.346808 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:33.351414 (+ 4606us) server_negotiation.cc:407] Connection header received
0620 09:30:33.351615 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:33.351617 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:33.351639 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:33.351673 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:30:33.352661 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:33.353397 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:33.355004 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:33.355309 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:33.355350 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:33.357836 (+ 2486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:30:33.357837 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:30:33.357838 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:30:33.357846 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:30:33.359479 (+ 1633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:30:33.360988 (+ 1509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:30:33.360989 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:30:33.360989 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:30:33.361048 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:30:33.361375 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:30:33.361376 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:30:33.361377 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:30:33.361592 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:30:33.361666 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:33.362121 (+ 455us) server_negotiation.cc:299] Negotiation successful
0620 09:30:33.362210 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 09:30:33.362713 6533 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25cf6661c27944438315b0dacb2037a1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 33800f9b25234b5f90881a0b27c6c866. Latest consensus state: current_term: 1 leader_uuid: "33800f9b25234b5f90881a0b27c6c866" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33800f9b25234b5f90881a0b27c6c866" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 37413 } } peers { permanent_uuid: "bf7dd9dca41e4440b8f1c6e7809d57ef" member_type: VOTER last_known_addr { host: "127.16.43.189" port: 37219 } } peers { permanent_uuid: "25cf6661c27944438315b0dacb2037a1" member_type: VOTER last_known_addr { host: "127.16.43.188" port: 42959 } } }
I20260620 09:30:33.362811 6533 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25cf6661c27944438315b0dacb2037a1 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 09:30:33.363919 6547 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33800f9b25234b5f90881a0b27c6c866 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "33800f9b25234b5f90881a0b27c6c866" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33800f9b25234b5f90881a0b27c6c866" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 37413 } } peers { permanent_uuid: "bf7dd9dca41e4440b8f1c6e7809d57ef" member_type: VOTER last_known_addr { host: "127.16.43.189" port: 37219 } } peers { permanent_uuid: "25cf6661c27944438315b0dacb2037a1" member_type: VOTER last_known_addr { host: "127.16.43.188" port: 42959 } } }
I20260620 09:30:33.364012 6547 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33800f9b25234b5f90881a0b27c6c866 [sys.catalog]: This master's current role is: LEADER
I20260620 09:30:33.364343 6533 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25cf6661c27944438315b0dacb2037a1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "33800f9b25234b5f90881a0b27c6c866" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33800f9b25234b5f90881a0b27c6c866" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 37413 } } peers { permanent_uuid: "bf7dd9dca41e4440b8f1c6e7809d57ef" member_type: VOTER last_known_addr { host: "127.16.43.189" port: 37219 } } peers { permanent_uuid: "25cf6661c27944438315b0dacb2037a1" member_type: VOTER last_known_addr { host: "127.16.43.188" port: 42959 } } }
I20260620 09:30:33.364418 6533 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25cf6661c27944438315b0dacb2037a1 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 09:30:33.364571 6567 catalog_manager.cc:1347] Generated new cluster ID: f6e171d4959c4729942e9d82c4aa22ee
I20260620 09:30:33.364619 6567 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:30:33.364786 6568 mvcc.cc:205] Tried to move back new op lower bound from 7298858325442301952 to 7298858325375762432. Current Snapshot: MvccSnapshot[applied={T|T < 7298858325442301952 or (T in {7298858325442301952})}]
I20260620 09:30:33.367969 6571 mvcc.cc:205] Tried to move back new op lower bound from 7298858325442301952 to 7298858325375762432. Current Snapshot: MvccSnapshot[applied={T|T < 7298858325442301952}]
I20260620 09:30:33.371145 6452 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.43.188" port: 42959 } attempt: 3
Jun 20 09:30:33 dist-test-slave-706q krb5kdc[6296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947833, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.189@KRBTEST.COM for kudu/127.16.43.188@KRBTEST.COM
I20260620 09:30:33.379427 6454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:33.371433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.188:42959 (local address 127.0.0.1:43538)
0620 09:30:33.371541 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0620 09:30:33.371553 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 09:30:33.371594 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:30:33.371858 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:30:33.371859 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:30:33.372054 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:30:33.372105 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:30:33.372107 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:30:33.372972 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:30:33.372973 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:30:33.373792 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:30:33.373794 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:30:33.374225 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:30:33.374227 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:30:33.374287 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:33.374761 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:30:33.374767 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:30:33.376272 (+ 1505us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:30:33.378333 (+ 2061us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:30:33.378334 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:30:33.378335 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:30:33.378546 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:30:33.378759 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:30:33.378760 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:30:33.378760 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:30:33.378820 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:30:33.379108 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:30:33.379109 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:30:33.379162 (+ 53us) client_negotiation.cc:769] Sending connection context
0620 09:30:33.379227 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 09:30:33.379324 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260620 09:30:33.379424 6531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:33.371691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43538 (local address 127.16.43.188:42959)
0620 09:30:33.371730 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:33.371732 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:33.371739 (+ 7us) server_negotiation.cc:407] Connection header received
0620 09:30:33.371759 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:33.371760 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:33.371768 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:33.371814 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:30:33.372175 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:33.372904 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:33.373871 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:33.374171 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:33.374227 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:33.376374 (+ 2147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:30:33.376375 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:30:33.376376 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:30:33.376383 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:30:33.378236 (+ 1853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:30:33.378620 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:30:33.378621 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:30:33.378622 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:30:33.378705 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:30:33.378892 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:30:33.378893 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:30:33.378894 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:30:33.379044 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:30:33.379110 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:33.379248 (+ 138us) server_negotiation.cc:299] Negotiation successful
0620 09:30:33.379327 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 09:30:33.381110 6452 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf7dd9dca41e4440b8f1c6e7809d57ef: Bootstrap starting.
I20260620 09:30:33.381866 6452 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bf7dd9dca41e4440b8f1c6e7809d57ef: Neither blocks nor log segments found. Creating new log.
I20260620 09:30:33.382180 6452 log.cc:824] T 00000000000000000000000000000000 P bf7dd9dca41e4440b8f1c6e7809d57ef: Log is configured to *not* fsync() on all Append() calls
I20260620 09:30:33.383122 6452 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf7dd9dca41e4440b8f1c6e7809d57ef: No bootstrap required, opened a new log
I20260620 09:30:33.386278 6452 raft_consensus.cc:348] T 00000000000000000000000000000000 P bf7dd9dca41e4440b8f1c6e7809d57ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33800f9b25234b5f90881a0b27c6c866" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 37413 } } peers { permanent_uuid: "bf7dd9dca41e4440b8f1c6e7809d57ef" member_type: VOTER last_known_addr { host: "127.16.43.189" port: 37219 } } peers { permanent_uuid: "25cf6661c27944438315b0dacb2037a1" member_type: VOTER last_known_addr { host: "127.16.43.188" port: 42959 } }
I20260620 09:30:33.386581 6452 raft_consensus.cc:729] T 00000000000000000000000000000000 P bf7dd9dca41e4440b8f1c6e7809d57ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf7dd9dca41e4440b8f1c6e7809d57ef, State: Initialized, Role: FOLLOWER
I20260620 09:30:33.386801 6452 consensus_queue.cc:260] T 00000000000000000000000000000000 P bf7dd9dca41e4440b8f1c6e7809d57ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33800f9b25234b5f90881a0b27c6c866" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 37413 } } peers { permanent_uuid: "bf7dd9dca41e4440b8f1c6e7809d57ef" member_type: VOTER last_known_addr { host: "127.16.43.189" port: 37219 } } peers { permanent_uuid: "25cf6661c27944438315b0dacb2037a1" member_type: VOTER last_known_addr { host: "127.16.43.188" port: 42959 } }
I20260620 09:30:33.388474 6426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "33800f9b25234b5f90881a0b27c6c866" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "bf7dd9dca41e4440b8f1c6e7809d57ef"
I20260620 09:30:33.388532 6425 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bf7dd9dca41e4440b8f1c6e7809d57ef [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:30:33.389596 6425 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bf7dd9dca41e4440b8f1c6e7809d57ef [term 1 FOLLOWER]: Refusing update from remote peer 33800f9b25234b5f90881a0b27c6c866: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:30:33.389959 6565 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 33800f9b25234b5f90881a0b27c6c866 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf7dd9dca41e4440b8f1c6e7809d57ef" member_type: VOTER last_known_addr { host: "127.16.43.189" port: 37219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:30:33.390898 6426 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bf7dd9dca41e4440b8f1c6e7809d57ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33800f9b25234b5f90881a0b27c6c866 in term 1.
I20260620 09:30:33.388422 6452 sys_catalog.cc:564] T 00000000000000000000000000000000 P bf7dd9dca41e4440b8f1c6e7809d57ef [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:30:33.393045 6567 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:30:33.393854 6567 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:30:33.395630 6576 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf7dd9dca41e4440b8f1c6e7809d57ef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "33800f9b25234b5f90881a0b27c6c866" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33800f9b25234b5f90881a0b27c6c866" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 37413 } } peers { permanent_uuid: "bf7dd9dca41e4440b8f1c6e7809d57ef" member_type: VOTER last_known_addr { host: "127.16.43.189" port: 37219 } } peers { permanent_uuid: "25cf6661c27944438315b0dacb2037a1" member_type: VOTER last_known_addr { host: "127.16.43.188" port: 42959 } } }
I20260620 09:30:33.395725 6576 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf7dd9dca41e4440b8f1c6e7809d57ef [sys.catalog]: This master's current role is: FOLLOWER
I20260620 09:30:33.395780 6576 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf7dd9dca41e4440b8f1c6e7809d57ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 33800f9b25234b5f90881a0b27c6c866. Latest consensus state: current_term: 1 leader_uuid: "33800f9b25234b5f90881a0b27c6c866" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33800f9b25234b5f90881a0b27c6c866" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 37413 } } peers { permanent_uuid: "bf7dd9dca41e4440b8f1c6e7809d57ef" member_type: VOTER last_known_addr { host: "127.16.43.189" port: 37219 } } peers { permanent_uuid: "25cf6661c27944438315b0dacb2037a1" member_type: VOTER last_known_addr { host: "127.16.43.188" port: 42959 } } }
I20260620 09:30:33.395819 6576 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf7dd9dca41e4440b8f1c6e7809d57ef [sys.catalog]: This master's current role is: FOLLOWER
I20260620 09:30:33.395853 6576 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf7dd9dca41e4440b8f1c6e7809d57ef [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "33800f9b25234b5f90881a0b27c6c866" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33800f9b25234b5f90881a0b27c6c866" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 37413 } } peers { permanent_uuid: "bf7dd9dca41e4440b8f1c6e7809d57ef" member_type: VOTER last_known_addr { host: "127.16.43.189" port: 37219 } } peers { permanent_uuid: "25cf6661c27944438315b0dacb2037a1" member_type: VOTER last_known_addr { host: "127.16.43.188" port: 42959 } } }
I20260620 09:30:33.395898 6576 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf7dd9dca41e4440b8f1c6e7809d57ef [sys.catalog]: This master's current role is: FOLLOWER
I20260620 09:30:33.400171 6581 mvcc.cc:205] Tried to move back new op lower bound from 7298858325442301952 to 7298858325375762432. Current Snapshot: MvccSnapshot[applied={T|T < 7298858325442301952}]
I20260620 09:30:33.401791 6567 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 33800f9b25234b5f90881a0b27c6c866: Generated new TSK 0
I20260620 09:30:33.401983 6567 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 09:30:33.402539 6591 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bf7dd9dca41e4440b8f1c6e7809d57ef: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260620 09:30:33.402940 6565 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33800f9b25234b5f90881a0b27c6c866 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "33800f9b25234b5f90881a0b27c6c866" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33800f9b25234b5f90881a0b27c6c866" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 37413 } } peers { permanent_uuid: "bf7dd9dca41e4440b8f1c6e7809d57ef" member_type: VOTER last_known_addr { host: "127.16.43.189" port: 37219 } } peers { permanent_uuid: "25cf6661c27944438315b0dacb2037a1" member_type: VOTER last_known_addr { host: "127.16.43.188" port: 42959 } } }
I20260620 09:30:33.403031 6565 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33800f9b25234b5f90881a0b27c6c866 [sys.catalog]: This master's current role is: LEADER
W20260620 09:30:33.404057 6591 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260620 09:30:33.427073 6456 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:30:33.425187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41402 (local address 127.16.43.189:37219)
0620 09:30:33.425235 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:33.425238 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:33.426382 (+ 1144us) server_negotiation.cc:407] Connection header received
0620 09:30:33.426492 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:33.426494 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:33.426506 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:33.426537 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:30:33.426998 (+ 461us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41402: BlockingRecv error: recv got EOF from 127.0.0.1:41402 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260620 09:30:33.427852 6380 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:30:33.425777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36360 (local address 127.16.43.190:37413)
0620 09:30:33.425818 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:33.425820 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:33.427315 (+ 1495us) server_negotiation.cc:407] Connection header received
0620 09:30:33.427417 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:33.427418 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:33.427430 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:33.427459 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:30:33.427791 (+ 332us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36360: BlockingRecv error: recv got EOF from 127.0.0.1:36360 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260620 09:30:33.428598 6531 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:30:33.426125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43542 (local address 127.16.43.188:42959)
0620 09:30:33.426164 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:33.426165 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:33.428060 (+ 1895us) server_negotiation.cc:407] Connection header received
0620 09:30:33.428209 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:33.428210 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:33.428221 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:33.428252 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:30:33.428540 (+ 288us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43542: BlockingRecv error: recv got EOF from 127.0.0.1:43542 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260620 09:30:33.437320 6456 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:30:33.435884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41410 (local address 127.16.43.189:37219)
0620 09:30:33.435931 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:33.435933 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:33.436684 (+ 751us) server_negotiation.cc:407] Connection header received
0620 09:30:33.436811 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:33.436812 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:33.436833 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:33.436865 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:30:33.437260 (+ 395us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41410: BlockingRecv error: recv got EOF from 127.0.0.1:41410 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260620 09:30:33.440763 6456 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:30:33.439868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41420 (local address 127.16.43.189:37219)
0620 09:30:33.439903 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:33.439905 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:33.440166 (+ 261us) server_negotiation.cc:407] Connection header received
0620 09:30:33.440272 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:33.440273 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:33.440285 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:33.440315 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:30:33.440678 (+ 363us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41420: BlockingRecv error: recv got EOF from 127.0.0.1:41420 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260620 09:30:33.444826 6456 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:30:33.444004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41428 (local address 127.16.43.189:37219)
0620 09:30:33.444041 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:33.444043 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:33.444296 (+ 253us) server_negotiation.cc:407] Connection header received
0620 09:30:33.444397 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:33.444399 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:33.444410 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:33.444438 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:30:33.444774 (+ 336us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41428: BlockingRecv error: recv got EOF from 127.0.0.1:41428 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260620 09:30:33.450834 6456 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:30:33.449925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41444 (local address 127.16.43.189:37219)
0620 09:30:33.449973 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:33.449975 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:33.450246 (+ 271us) server_negotiation.cc:407] Connection header received
0620 09:30:33.450352 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:33.450353 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:33.450364 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:33.450422 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:30:33.450783 (+ 361us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41444: BlockingRecv error: recv got EOF from 127.0.0.1:41444 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
W20260620 09:30:33.461938 6456 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:30:33.460967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41456 (local address 127.16.43.189:37219)
0620 09:30:33.461015 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:33.461018 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:33.461324 (+ 306us) server_negotiation.cc:407] Connection header received
0620 09:30:33.461436 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:33.461437 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:33.461449 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:33.461482 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:30:33.461873 (+ 391us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41456: BlockingRecv error: recv got EOF from 127.0.0.1:41456 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260620 09:30:33.480696 6456 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:30:33.479276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41458 (local address 127.16.43.189:37219)
0620 09:30:33.479322 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:33.479324 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:33.479980 (+ 656us) server_negotiation.cc:407] Connection header received
0620 09:30:33.480103 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:33.480105 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:33.480117 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:33.480147 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:30:33.480613 (+ 466us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41458: BlockingRecv error: recv got EOF from 127.0.0.1:41458 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260620 09:30:33.514883 6456 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:30:33.513988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41466 (local address 127.16.43.189:37219)
0620 09:30:33.514036 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:33.514038 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:33.514320 (+ 282us) server_negotiation.cc:407] Connection header received
0620 09:30:33.514419 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:33.514421 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:33.514432 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:33.514464 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:30:33.514826 (+ 362us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41466: BlockingRecv error: recv got EOF from 127.0.0.1:41466 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260620 09:30:33.581091 6456 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:30:33.580235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41470 (local address 127.16.43.189:37219)
0620 09:30:33.580280 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:33.580282 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:33.580384 (+ 102us) server_negotiation.cc:407] Connection header received
0620 09:30:33.580444 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:33.580445 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:33.580510 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:33.580578 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:30:33.581016 (+ 438us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41470: BlockingRecv error: recv got EOF from 127.0.0.1:41470 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260620 09:30:33.711197 6456 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:30:33.710403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41486 (local address 127.16.43.189:37219)
0620 09:30:33.710492 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:33.710495 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:33.710584 (+ 89us) server_negotiation.cc:407] Connection header received
0620 09:30:33.710690 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:33.710691 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:33.710702 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:33.710772 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:30:33.711111 (+ 339us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41486: BlockingRecv error: recv got EOF from 127.0.0.1:41486 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260620 09:30:33.969339 6456 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:30:33.968653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41500 (local address 127.16.43.189:37219)
0620 09:30:33.968728 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:33.968730 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:33.968742 (+ 12us) server_negotiation.cc:407] Connection header received
0620 09:30:33.968780 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:33.968782 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:33.968794 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:33.968869 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:30:33.969244 (+ 375us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41500: BlockingRecv error: recv got EOF from 127.0.0.1:41500 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 09:30:34.332554 6544 catalog_manager.cc:1259] Loaded cluster ID: f6e171d4959c4729942e9d82c4aa22ee
I20260620 09:30:34.332671 6544 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 25cf6661c27944438315b0dacb2037a1: loading cluster ID for follower catalog manager: success
I20260620 09:30:34.333470 6544 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 25cf6661c27944438315b0dacb2037a1: acquiring CA information for follower catalog manager: success
I20260620 09:30:34.333809 6544 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 25cf6661c27944438315b0dacb2037a1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 09:30:34.404549 6591 catalog_manager.cc:1259] Loaded cluster ID: f6e171d4959c4729942e9d82c4aa22ee
I20260620 09:30:34.404640 6591 catalog_manager.cc:1552] T 00000000000000000000000000000000 P bf7dd9dca41e4440b8f1c6e7809d57ef: loading cluster ID for follower catalog manager: success
I20260620 09:30:34.405457 6591 catalog_manager.cc:1574] T 00000000000000000000000000000000 P bf7dd9dca41e4440b8f1c6e7809d57ef: acquiring CA information for follower catalog manager: success
I20260620 09:30:34.405783 6591 catalog_manager.cc:1602] T 00000000000000000000000000000000 P bf7dd9dca41e4440b8f1c6e7809d57ef: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 09:30:34.485539 6660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:34.482538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41516 (local address 127.16.43.189:37219)
0620 09:30:34.482664 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:34.482666 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:34.482754 (+ 88us) server_negotiation.cc:407] Connection header received
0620 09:30:34.482846 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:34.482847 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:34.482925 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:34.483026 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:30:34.483185 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:34.483899 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:34.484690 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:34.484896 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:34.484977 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:34.485169 (+ 192us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:30:34.485279 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:30:34.485373 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:34.485406 (+ 33us) server_negotiation.cc:299] Negotiation successful
0620 09:30:34.485431 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":55,"threads_started":1}
I20260620 09:30:34.489830 6666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:34.487573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43550 (local address 127.16.43.188:42959)
0620 09:30:34.487707 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:34.487709 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:34.487717 (+ 8us) server_negotiation.cc:407] Connection header received
0620 09:30:34.487753 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:34.487753 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:34.487790 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:34.487868 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:30:34.487953 (+ 85us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:34.488436 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:34.489158 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:34.489369 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:34.489441 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:34.489548 (+ 107us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:30:34.489654 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:30:34.489712 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:34.489744 (+ 32us) server_negotiation.cc:299] Negotiation successful
0620 09:30:34.489764 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":83,"threads_started":1}
I20260620 09:30:34.495066 6666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:34.492029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43560 (local address 127.16.43.188:42959)
0620 09:30:34.492081 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:34.492083 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:34.492245 (+ 162us) server_negotiation.cc:407] Connection header received
0620 09:30:34.492366 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:34.492367 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:34.492379 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:34.492408 (+ 29us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:30:34.492579 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:34.493062 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:34.494214 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:34.494518 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:34.494591 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:34.494741 (+ 150us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:30:34.494769 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:30:34.494836 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:34.494953 (+ 117us) server_negotiation.cc:299] Negotiation successful
0620 09:30:34.494987 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 09:30:34.495180 6660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:34.491855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41518 (local address 127.16.43.189:37219)
0620 09:30:34.491899 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:34.491901 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:34.492222 (+ 321us) server_negotiation.cc:407] Connection header received
0620 09:30:34.492272 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:34.492274 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:34.492285 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:34.492335 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:30:34.492513 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:34.493244 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:34.494214 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:34.494518 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:34.494590 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:34.494885 (+ 295us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:30:34.494923 (+ 38us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:30:34.494970 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:34.495054 (+ 84us) server_negotiation.cc:299] Negotiation successful
0620 09:30:34.495078 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 09:30:34.496097 6672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:34.491777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36362 (local address 127.16.43.190:37413)
0620 09:30:34.492069 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:34.492071 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:34.492081 (+ 10us) server_negotiation.cc:407] Connection header received
0620 09:30:34.492129 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:34.492130 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:34.492186 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:34.492310 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:30:34.492446 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:34.493235 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:34.495041 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:34.495363 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:34.495458 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:34.495767 (+ 309us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:30:34.495857 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:30:34.495937 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:34.495984 (+ 47us) server_negotiation.cc:299] Negotiation successful
0620 09:30:34.496010 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":61,"threads_started":1}
I20260620 09:30:34.498036 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 6309
I20260620 09:30:34.503918 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 6384
I20260620 09:30:34.510639 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 6459
2026-06-20T09:30:34Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2846 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781947562949448-16558-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:30:34 dist-test-slave-706q krb5kdc[6677](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:30:34 dist-test-slave-706q krb5kdc[6677](info): set up 2 sockets
Jun 20 09:30:34 dist-test-slave-706q krb5kdc[6677](info): commencing operation
krb5kdc: starting...
W20260620 09:30:35.556835 16558 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781947562949448-16558-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781947562949448-16558-0/krb5kdc/test-user.keytab.
Jun 20 09:30:35 dist-test-slave-706q krb5kdc[6677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947835, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:30:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:30:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.43.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.43.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.43.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.190.keytab.
Entry for principal kudu/127.16.43.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.190.keytab.
I20260620 09:30:35.680213 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781947562949448-16558-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781947562949448-16558-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781947562949448-16558-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.43.190:34499
--webserver_interface=127.16.43.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.190.keytab
--principal=kudu/127.16.43.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.43.148:39391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.43.190:34499,127.16.43.189:38581,127.16.43.188:44961
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781947562949448-16558-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781947562949448-16558-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781947562949448-16558-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:30:35.762729 6690 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 09:30:35.762969 6690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:30:35.762987 6690 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:30:35.763010 6690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:30:35.764171 6690 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:30:35.764210 6690 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:30:35.764225 6690 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:30:35.764236 6690 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:30:35.764249 6690 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:30:35.765892 6690 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:39391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781947562949448-16558-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781947562949448-16558-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.43.190:34499,127.16.43.189:38581,127.16.43.188:44961
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.190.keytab
--principal=kudu/127.16.43.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.43.190:34499
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.43.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6690
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781947562949448-16558-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:30:35.766211 6690 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:30:35.766458 6690 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 09:30:35.771823 6690 server_base.cc:1034] running on GCE node
W20260620 09:30:35.771786 6696 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:30:35.771781 6695 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:30:35.771798 6698 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:30:35.772284 6690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:30:35.772550 6690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:30:35.773705 6690 hybrid_clock.cc:648] HybridClock initialized: now 1781947835773685 us; error 42 us; skew 500 ppm
Jun 20 09:30:35 dist-test-slave-706q krb5kdc[6677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947835, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:30:35.776463 6690 init.cc:377] Logged in from keytab as kudu/127.16.43.190@KRBTEST.COM (short username kudu)
I20260620 09:30:35.776928 6690 webserver.cc:460] Webserver started at http://127.16.43.190:42963/ using document root <none> and password file <none>
I20260620 09:30:35.777159 6690 fs_manager.cc:359] Metadata directory not provided
I20260620 09:30:35.777230 6690 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:30:35.777385 6690 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:30:35.778514 6690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781947562949448-16558-0/minicluster-data/master-0/data/instance:
uuid: "1a15b670a7b84fdf832af1cdbc1cf070"
format_stamp: "Formatted at 2026-06-20 09:30:35 on dist-test-slave-706q"
server_key: "81ec3579151c822075cb1de1653f66f5"
server_key_iv: "a3e28d5ed05ab287e7dbb15b43c0f501"
server_key_version: "encryptionkey@0"
I20260620 09:30:35.778910 6690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781947562949448-16558-0/minicluster-data/master-0/wal/instance:
uuid: "1a15b670a7b84fdf832af1cdbc1cf070"
format_stamp: "Formatted at 2026-06-20 09:30:35 on dist-test-slave-706q"
server_key: "81ec3579151c822075cb1de1653f66f5"
server_key_iv: "a3e28d5ed05ab287e7dbb15b43c0f501"
server_key_version: "encryptionkey@0"
I20260620 09:30:35.780340 6690 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 09:30:35.781131 6705 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:30:35.781314 6690 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 09:30:35.781416 6690 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781947562949448-16558-0/minicluster-data/master-0/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781947562949448-16558-0/minicluster-data/master-0/wal
uuid: "1a15b670a7b84fdf832af1cdbc1cf070"
format_stamp: "Formatted at 2026-06-20 09:30:35 on dist-test-slave-706q"
server_key: "81ec3579151c822075cb1de1653f66f5"
server_key_iv: "a3e28d5ed05ab287e7dbb15b43c0f501"
server_key_version: "encryptionkey@0"
I20260620 09:30:35.781499 6690 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781947562949448-16558-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781947562949448-16558-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781947562949448-16558-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:30:35.805375 6690 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:30:35.808151 6690 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:30:35.808274 6690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:30:35.812193 6690 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.190:34499
I20260620 09:30:35.812224 6757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.190:34499 every 8 connection(s)
I20260620 09:30:35.812597 6690 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781947562949448-16558-0/minicluster-data/master-0/data/info.pb
I20260620 09:30:35.813593 6758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:30:35.815102 6758 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.43.190" port: 34499 } has no permanent_uuid. Determining permanent_uuid...
I20260620 09:30:35.815604 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 6690
I20260620 09:30:35.815729 16558 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781947562949448-16558-0/minicluster-data/master-0/wal/instance
I20260620 09:30:35.815964 16558 external_mini_cluster.cc:1428] Setting key abc61f533f36a80a5fe137cb4f154cdf
Jun 20 09:30:35 dist-test-slave-706q krb5kdc[6677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947835, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.190@KRBTEST.COM for kudu/127.16.43.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.43.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.43.189@KRBTEST.COM" created.
I20260620 09:30:35.826558 6760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:35.815825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:34499 (local address 127.0.0.1:45600)
0620 09:30:35.816007 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0620 09:30:35.816048 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 09:30:35.816505 (+ 457us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:30:35.817576 (+ 1071us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:30:35.817580 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:30:35.817967 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:30:35.818210 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:30:35.818216 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:30:35.819252 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:30:35.819253 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:30:35.820024 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:30:35.820027 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:30:35.820589 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:30:35.820590 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:30:35.820653 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:35.821217 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:30:35.821225 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:30:35.822860 (+ 1635us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:30:35.824725 (+ 1865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:30:35.824731 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:30:35.824733 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:30:35.824950 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:30:35.825268 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:30:35.825269 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:30:35.825269 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:30:35.825355 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:30:35.826006 (+ 651us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:30:35.826013 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:30:35.826114 (+ 101us) client_negotiation.cc:769] Sending connection context
0620 09:30:35.826217 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 09:30:35.826380 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":107,"threads_started":1}
I20260620 09:30:35.826653 6761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:35.815844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45600 (local address 127.16.43.190:34499)
0620 09:30:35.817312 (+ 1468us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:35.817317 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:35.817343 (+ 26us) server_negotiation.cc:407] Connection header received
0620 09:30:35.817422 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:35.817423 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:35.817440 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:35.817523 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:30:35.818318 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:35.819140 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:35.820124 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:35.820467 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:35.820607 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:35.822954 (+ 2347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:30:35.822964 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:30:35.822964 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:30:35.822973 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:30:35.824624 (+ 1651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:30:35.825097 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:30:35.825100 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:30:35.825100 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:30:35.825179 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:30:35.825468 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:30:35.825469 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:30:35.825470 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:30:35.825870 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:30:35.825998 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:35.826312 (+ 314us) server_negotiation.cc:299] Negotiation successful
0620 09:30:35.826472 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":42,"server-negotiator.queue_time_us":1433,"thread_start_us":104,"threads_started":1}
I20260620 09:30:35.827407 6758 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.43.189" port: 38581 } has no permanent_uuid. Determining permanent_uuid...
I20260620 09:30:35.827836 6760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:35.827737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.189:38581 (local address 127.0.0.1:42884)
0620 09:30:35.827773 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0620 09:30:35.827816 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.189:38581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20260620 09:30:35.828001 6706 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.43.189:38581: connect: Connection refused (error 111)
I20260620 09:30:35.828456 6760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:35.828352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.189:38581 (local address 127.0.0.1:42888)
0620 09:30:35.828405 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0620 09:30:35.828431 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.189:38581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260620 09:30:35.828583 6758 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.43.189:38581: Network error: Client connection negotiation failed: client connection to 127.16.43.189:38581: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.43.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.189.keytab.
Entry for principal kudu/127.16.43.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.189.keytab.
I20260620 09:30:35.843116 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781947562949448-16558-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781947562949448-16558-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781947562949448-16558-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781947562949448-16558-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.43.189:38581
--webserver_interface=127.16.43.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.189.keytab
--principal=kudu/127.16.43.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.43.148:39391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.43.190:34499,127.16.43.189:38581,127.16.43.188:44961
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781947562949448-16558-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781947562949448-16558-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781947562949448-16558-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 09:30:35.877838 6758 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.43.189" port: 38581 } attempt: 1
I20260620 09:30:35.878441 6760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:35.878259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.189:38581 (local address 127.0.0.1:42902)
0620 09:30:35.878336 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0620 09:30:35.878394 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.189:38581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260620 09:30:35.879097 6760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:35.878961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.189:38581 (local address 127.0.0.1:42916)
0620 09:30:35.879024 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0620 09:30:35.879063 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.189:38581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260620 09:30:35.879256 6758 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.43.189:38581: Network error: Client connection negotiation failed: client connection to 127.16.43.189:38581: connect: Connection refused (error 111)
W20260620 09:30:35.925118 6765 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 09:30:35.925367 6765 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:30:35.925385 6765 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:30:35.925407 6765 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:30:35.926765 6765 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:30:35.926824 6765 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:30:35.926838 6765 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:30:35.926852 6765 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:30:35.926864 6765 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:30:35.928571 6765 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:39391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781947562949448-16558-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781947562949448-16558-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.16.43.190:34499,127.16.43.189:38581,127.16.43.188:44961
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.189.keytab
--principal=kudu/127.16.43.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.43.189:38581
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781947562949448-16558-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.16.43.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6765
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781947562949448-16558-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:30:35.928802 6765 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:30:35.928982 6765 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 09:30:35.934227 6765 server_base.cc:1034] running on GCE node
W20260620 09:30:35.934154 6773 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:30:35.934154 6771 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:30:35.934149 6770 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:30:35.934644 6765 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:30:35.935009 6765 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:30:35.936187 6765 hybrid_clock.cc:648] HybridClock initialized: now 1781947835936160 us; error 45 us; skew 500 ppm
Jun 20 09:30:35 dist-test-slave-706q krb5kdc[6677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947835, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:30:35.938977 6765 init.cc:377] Logged in from keytab as kudu/127.16.43.189@KRBTEST.COM (short username kudu)
I20260620 09:30:35.939498 6765 webserver.cc:460] Webserver started at http://127.16.43.189:45999/ using document root <none> and password file <none>
I20260620 09:30:35.939817 6765 fs_manager.cc:359] Metadata directory not provided
I20260620 09:30:35.939896 6765 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:30:35.940055 6765 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:30:35.941205 6765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781947562949448-16558-0/minicluster-data/master-1/data/instance:
uuid: "fedac94dd8c8486c939f8f9ee52ba284"
format_stamp: "Formatted at 2026-06-20 09:30:35 on dist-test-slave-706q"
server_key: "977fd2f993840972745ddb859c1a2e58"
server_key_iv: "81f30ae1cafa46850a66925f4fdd2362"
server_key_version: "encryptionkey@0"
I20260620 09:30:35.941596 6765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781947562949448-16558-0/minicluster-data/master-1/wal/instance:
uuid: "fedac94dd8c8486c939f8f9ee52ba284"
format_stamp: "Formatted at 2026-06-20 09:30:35 on dist-test-slave-706q"
server_key: "977fd2f993840972745ddb859c1a2e58"
server_key_iv: "81f30ae1cafa46850a66925f4fdd2362"
server_key_version: "encryptionkey@0"
I20260620 09:30:35.942988 6765 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 09:30:35.943946 6780 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:30:35.944134 6765 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 09:30:35.944239 6765 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781947562949448-16558-0/minicluster-data/master-1/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781947562949448-16558-0/minicluster-data/master-1/wal
uuid: "fedac94dd8c8486c939f8f9ee52ba284"
format_stamp: "Formatted at 2026-06-20 09:30:35 on dist-test-slave-706q"
server_key: "977fd2f993840972745ddb859c1a2e58"
server_key_iv: "81f30ae1cafa46850a66925f4fdd2362"
server_key_version: "encryptionkey@0"
I20260620 09:30:35.944326 6765 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781947562949448-16558-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781947562949448-16558-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781947562949448-16558-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:30:35.947434 6758 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.43.189" port: 38581 } attempt: 2
I20260620 09:30:35.947875 6760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:35.947760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.189:38581 (local address 127.0.0.1:42926)
0620 09:30:35.947815 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0620 09:30:35.947844 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.189:38581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260620 09:30:35.948279 6760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:35.948181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.189:38581 (local address 127.0.0.1:42940)
0620 09:30:35.948234 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0620 09:30:35.948260 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.189:38581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260620 09:30:35.948410 6758 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.43.189:38581: Network error: Client connection negotiation failed: client connection to 127.16.43.189:38581: connect: Connection refused (error 111)
I20260620 09:30:35.958900 6765 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:30:35.961438 6765 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:30:35.961627 6765 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:30:35.965798 6765 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.189:38581
I20260620 09:30:35.965838 6832 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.189:38581 every 8 connection(s)
I20260620 09:30:35.966238 6765 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781947562949448-16558-0/minicluster-data/master-1/data/info.pb
I20260620 09:30:35.967226 6833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:30:35.968279 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 6765
I20260620 09:30:35.968407 16558 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781947562949448-16558-0/minicluster-data/master-1/wal/instance
I20260620 09:30:35.968654 16558 external_mini_cluster.cc:1428] Setting key bd55f8d3b9ae23585e77f1afb6300472
I20260620 09:30:35.969427 6833 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.43.190" port: 34499 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 09:30:35 dist-test-slave-706q krb5kdc[6677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947835, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.189@KRBTEST.COM for kudu/127.16.43.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.43.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.43.188@KRBTEST.COM" created.
I20260620 09:30:35.979621 6761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:35.970376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45602 (local address 127.16.43.190:34499)
0620 09:30:35.970437 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:35.970440 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:35.970648 (+ 208us) server_negotiation.cc:407] Connection header received
0620 09:30:35.971166 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:35.971167 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:35.971183 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:35.971233 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:30:35.971955 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:35.972535 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:35.973543 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:35.973772 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:35.973865 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:35.976134 (+ 2269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:30:35.976136 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:30:35.976136 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:30:35.976144 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:30:35.977517 (+ 1373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:30:35.977988 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:30:35.977993 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:30:35.977994 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:30:35.978046 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:30:35.978381 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:30:35.978382 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:30:35.978383 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:30:35.978679 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:30:35.978877 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:35.979404 (+ 527us) server_negotiation.cc:299] Negotiation successful
0620 09:30:35.979498 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 09:30:35.979764 6836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:35.970334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:34499 (local address 127.0.0.1:45602)
0620 09:30:35.970523 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0620 09:30:35.970559 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 09:30:35.971048 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:30:35.971300 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:30:35.971309 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:30:35.971650 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:30:35.971849 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:30:35.971853 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:30:35.972650 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:30:35.972651 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:30:35.973446 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:30:35.973448 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:30:35.973888 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:30:35.973889 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:30:35.974017 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:35.974593 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:30:35.974601 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:30:35.976019 (+ 1418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:30:35.977626 (+ 1607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:30:35.977634 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:30:35.977636 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:30:35.977889 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:30:35.978122 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:30:35.978122 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:30:35.978123 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:30:35.978303 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:30:35.978866 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:30:35.978874 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:30:35.979014 (+ 140us) client_negotiation.cc:769] Sending connection context
0620 09:30:35.979309 (+ 295us) client_negotiation.cc:240] Negotiation successful
0620 09:30:35.979506 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":88,"threads_started":1}
I20260620 09:30:35.980590 6833 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.43.189" port: 38581 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 09:30:35 dist-test-slave-706q krb5kdc[6677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947835, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.189@KRBTEST.COM for kudu/127.16.43.189@KRBTEST.COM
I20260620 09:30:35.989233 6838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:35.980938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42956 (local address 127.16.43.189:38581)
0620 09:30:35.981068 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:35.981073 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:35.981090 (+ 17us) server_negotiation.cc:407] Connection header received
0620 09:30:35.981109 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:35.981111 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:35.981129 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:35.981188 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:30:35.981536 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:35.982206 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:35.983328 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:35.983936 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:35.984019 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:35.986089 (+ 2070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:30:35.986098 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:30:35.986098 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:30:35.986106 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:30:35.987956 (+ 1850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:30:35.988349 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:30:35.988352 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:30:35.988352 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:30:35.988412 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:30:35.988604 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:30:35.988605 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:30:35.988605 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:30:35.988751 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:30:35.988847 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:35.988981 (+ 134us) server_negotiation.cc:299] Negotiation successful
0620 09:30:35.989107 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":104,"thread_start_us":81,"threads_started":1}
I20260620 09:30:35.989239 6836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:35.980845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.189:38581 (local address 127.0.0.1:42956)
0620 09:30:35.980893 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0620 09:30:35.980903 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 09:30:35.980941 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:30:35.981225 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:30:35.981226 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:30:35.981392 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:30:35.981459 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:30:35.981461 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:30:35.982296 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:30:35.982297 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:30:35.983230 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:30:35.983233 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:30:35.984017 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:30:35.984018 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:30:35.984083 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:35.984545 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:30:35.984549 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:30:35.985998 (+ 1449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:30:35.988029 (+ 2031us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:30:35.988030 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:30:35.988030 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:30:35.988247 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:30:35.988481 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:30:35.988482 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:30:35.988482 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:30:35.988540 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:30:35.988824 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:30:35.988825 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:30:35.988863 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 09:30:35.988949 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 09:30:35.989156 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31,"mutex_wait_us":105}
I20260620 09:30:35.990041 6833 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.43.188" port: 44961 } has no permanent_uuid. Determining permanent_uuid...
I20260620 09:30:35.990541 6836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:35.990426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.188:44961 (local address 127.0.0.1:57314)
0620 09:30:35.990473 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0620 09:30:35.990523 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.188:44961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260620 09:30:35.990701 6781 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.43.188:44961: connect: Connection refused (error 111)
I20260620 09:30:35.991175 6836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:35.991117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.188:44961 (local address 127.0.0.1:57328)
0620 09:30:35.991140 (+ 23us) negotiation.cc:107] Waiting for socket to connect
0620 09:30:35.991161 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.188:44961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":13}
W20260620 09:30:35.991295 6833 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.43.188:44961: Network error: Client connection negotiation failed: client connection to 127.16.43.188:44961: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.43.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.188.keytab.
Entry for principal kudu/127.16.43.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.188.keytab.
I20260620 09:30:35.994479 16558 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMkdNwY/build/release/bin/kudu
/tmp/dist-test-taskMkdNwY/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781947562949448-16558-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781947562949448-16558-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781947562949448-16558-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781947562949448-16558-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.43.188:44961
--webserver_interface=127.16.43.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.188.keytab
--principal=kudu/127.16.43.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.43.148:39391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.43.190:34499,127.16.43.189:38581,127.16.43.188:44961
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781947562949448-16558-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781947562949448-16558-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781947562949448-16558-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 09:30:36.039631 6758 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.43.189" port: 38581 } attempt: 3
I20260620 09:30:36.040526 6833 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.43.188" port: 44961 } attempt: 1
I20260620 09:30:36.040899 6836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:36.040792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.188:44961 (local address 127.0.0.1:57338)
0620 09:30:36.040829 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0620 09:30:36.040866 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.188:44961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260620 09:30:36.041287 6836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:36.041169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.188:44961 (local address 127.0.0.1:57340)
0620 09:30:36.041236 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0620 09:30:36.041260 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.188:44961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260620 09:30:36.041464 6833 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.43.188:44961: Network error: Client connection negotiation failed: client connection to 127.16.43.188:44961: connect: Connection refused (error 111)
Jun 20 09:30:36 dist-test-slave-706q krb5kdc[6677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947835, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.190@KRBTEST.COM for kudu/127.16.43.189@KRBTEST.COM
I20260620 09:30:36.047737 6760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:36.040025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.189:38581 (local address 127.0.0.1:42964)
0620 09:30:36.040089 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0620 09:30:36.040103 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 09:30:36.040180 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:30:36.040318 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:30:36.040319 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:30:36.040598 (+ 279us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:30:36.040685 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:30:36.040688 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:30:36.041637 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:30:36.041638 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:30:36.042163 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:30:36.042164 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:30:36.042662 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:30:36.042663 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:30:36.042722 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:36.043210 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:30:36.043216 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:30:36.044702 (+ 1486us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:30:36.046596 (+ 1894us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:30:36.046597 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:30:36.046598 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:30:36.046787 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:30:36.047005 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:30:36.047005 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:30:36.047006 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:30:36.047059 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:30:36.047363 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:30:36.047364 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:30:36.047407 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 09:30:36.047466 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 09:30:36.047604 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260620 09:30:36.047849 6838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:36.040120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42964 (local address 127.16.43.189:38581)
0620 09:30:36.040199 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:36.040201 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:36.040213 (+ 12us) server_negotiation.cc:407] Connection header received
0620 09:30:36.040246 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:36.040248 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:36.040261 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:36.040315 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:30:36.040779 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:36.041553 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:36.042246 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:36.042584 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:36.042657 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:36.044807 (+ 2150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:30:36.044809 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:30:36.044809 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:30:36.044817 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:30:36.046452 (+ 1635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:30:36.046865 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:30:36.046866 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:30:36.046867 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:30:36.046932 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:30:36.047122 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:30:36.047123 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:30:36.047124 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:30:36.047292 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:30:36.047359 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:36.047653 (+ 294us) server_negotiation.cc:299] Negotiation successful
0620 09:30:36.047759 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 09:30:36.048559 6758 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.43.188" port: 44961 } has no permanent_uuid. Determining permanent_uuid...
I20260620 09:30:36.048940 6760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:36.048831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.188:44961 (local address 127.0.0.1:57342)
0620 09:30:36.048880 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0620 09:30:36.048909 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.188:44961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260620 09:30:36.049343 6760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:36.049262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.188:44961 (local address 127.0.0.1:57352)
0620 09:30:36.049307 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0620 09:30:36.049328 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.188:44961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260620 09:30:36.049459 6758 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.43.188:44961: Network error: Client connection negotiation failed: client connection to 127.16.43.188:44961: connect: Connection refused (error 111)
I20260620 09:30:36.080657 6758 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.43.188" port: 44961 } attempt: 1
I20260620 09:30:36.081162 6760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:36.081013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.188:44961 (local address 127.0.0.1:57368)
0620 09:30:36.081078 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0620 09:30:36.081127 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.188:44961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260620 09:30:36.081552 6760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:36.081443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.188:44961 (local address 127.0.0.1:57370)
0620 09:30:36.081493 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0620 09:30:36.081526 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.188:44961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260620 09:30:36.081449 6840 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 09:30:36.081691 6758 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.43.188:44961: Network error: Client connection negotiation failed: client connection to 127.16.43.188:44961: connect: Connection refused (error 111)
W20260620 09:30:36.081748 6840 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:30:36.081784 6840 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:30:36.081821 6840 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:30:36.082978 6840 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:30:36.083019 6840 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:30:36.083034 6840 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:30:36.083076 6840 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:30:36.083103 6840 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:30:36.084834 6840 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.43.148:39391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781947562949448-16558-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781947562949448-16558-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.16.43.190:34499,127.16.43.189:38581,127.16.43.188:44961
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781947562949448-16558-0/krb5kdc/kudu_127.16.43.188.keytab
--principal=kudu/127.16.43.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.43.188:44961
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781947562949448-16558-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.16.43.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6840
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781947562949448-16558-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:30:36.085119 6840 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:30:36.085384 6840 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:30:36.090459 6848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:30:36.090468 6846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:30:36.090559 6840 server_base.cc:1034] running on GCE node
W20260620 09:30:36.090463 6845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:30:36.091915 6840 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:30:36.092275 6840 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:30:36.093537 6840 hybrid_clock.cc:648] HybridClock initialized: now 1781947836093520 us; error 43 us; skew 500 ppm
Jun 20 09:30:36 dist-test-slave-706q krb5kdc[6677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947836, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:30:36.096184 6840 init.cc:377] Logged in from keytab as kudu/127.16.43.188@KRBTEST.COM (short username kudu)
I20260620 09:30:36.096679 6840 webserver.cc:460] Webserver started at http://127.16.43.188:38079/ using document root <none> and password file <none>
I20260620 09:30:36.096920 6840 fs_manager.cc:359] Metadata directory not provided
I20260620 09:30:36.096969 6840 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:30:36.097144 6840 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:30:36.098264 6840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781947562949448-16558-0/minicluster-data/master-2/data/instance:
uuid: "48ca9acf3b484e8fb17801e2a0126955"
format_stamp: "Formatted at 2026-06-20 09:30:36 on dist-test-slave-706q"
server_key: "5648ae3bbad77a82b9e1f2e000cce202"
server_key_iv: "b0cd1487804c82e04cc99eb7357a8e18"
server_key_version: "encryptionkey@0"
I20260620 09:30:36.098654 6840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781947562949448-16558-0/minicluster-data/master-2/wal/instance:
uuid: "48ca9acf3b484e8fb17801e2a0126955"
format_stamp: "Formatted at 2026-06-20 09:30:36 on dist-test-slave-706q"
server_key: "5648ae3bbad77a82b9e1f2e000cce202"
server_key_iv: "b0cd1487804c82e04cc99eb7357a8e18"
server_key_version: "encryptionkey@0"
I20260620 09:30:36.100148 6840 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 09:30:36.101081 6855 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:30:36.101322 6840 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:30:36.101493 6840 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781947562949448-16558-0/minicluster-data/master-2/data,/tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781947562949448-16558-0/minicluster-data/master-2/wal
uuid: "48ca9acf3b484e8fb17801e2a0126955"
format_stamp: "Formatted at 2026-06-20 09:30:36 on dist-test-slave-706q"
server_key: "5648ae3bbad77a82b9e1f2e000cce202"
server_key_iv: "b0cd1487804c82e04cc99eb7357a8e18"
server_key_version: "encryptionkey@0"
I20260620 09:30:36.101583 6840 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781947562949448-16558-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781947562949448-16558-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781947562949448-16558-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:30:36.109608 6833 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.43.188" port: 44961 } attempt: 2
I20260620 09:30:36.110064 6836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:36.109931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.188:44961 (local address 127.0.0.1:57380)
0620 09:30:36.109991 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0620 09:30:36.110032 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.188:44961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260620 09:30:36.110407 6836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:36.110317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.188:44961 (local address 127.0.0.1:57394)
0620 09:30:36.110361 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0620 09:30:36.110384 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.43.188:44961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260620 09:30:36.110523 6833 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.43.188:44961: Network error: Client connection negotiation failed: client connection to 127.16.43.188:44961: connect: Connection refused (error 111)
I20260620 09:30:36.118692 6840 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:30:36.121013 6840 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:30:36.121201 6840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:30:36.125563 6840 rpc_server.cc:307] RPC server started. Bound to: 127.16.43.188:44961
I20260620 09:30:36.125597 6907 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.43.188:44961 every 8 connection(s)
I20260620 09:30:36.126042 6840 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781947562949448-16558-0/minicluster-data/master-2/data/info.pb
I20260620 09:30:36.127012 6908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:30:36.128643 6908 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.43.190" port: 34499 } has no permanent_uuid. Determining permanent_uuid...
I20260620 09:30:36.129783 16558 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMkdNwY/build/release/bin/kudu as pid 6840
I20260620 09:30:36.129914 16558 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMkdNwY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781947562949448-16558-0/minicluster-data/master-2/wal/instance
I20260620 09:30:36.130137 16558 external_mini_cluster.cc:1428] Setting key 7c62841190fd50a893cbd8ca2ae6c828
Jun 20 09:30:36 dist-test-slave-706q krb5kdc[6677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947836, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.188@KRBTEST.COM for kudu/127.16.43.190@KRBTEST.COM
Jun 20 09:30:36 dist-test-slave-706q krb5kdc[6677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947835, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.43.188@KRBTEST.COM
I20260620 09:30:36.140182 6761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:36.129419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45606 (local address 127.16.43.190:34499)
0620 09:30:36.129555 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:36.129557 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:36.129631 (+ 74us) server_negotiation.cc:407] Connection header received
0620 09:30:36.130205 (+ 574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:36.130207 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:36.130219 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:36.130290 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:30:36.131195 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:36.131929 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:36.133159 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:36.133447 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:36.133521 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:36.136277 (+ 2756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:30:36.136279 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:30:36.136279 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:30:36.136288 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:30:36.138456 (+ 2168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:30:36.138986 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:30:36.138987 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:30:36.138988 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:30:36.139079 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:30:36.139344 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:30:36.139346 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:30:36.139346 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:30:36.139501 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:30:36.139604 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:36.139957 (+ 353us) server_negotiation.cc:299] Negotiation successful
0620 09:30:36.140064 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260620 09:30:36.140350 6912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:36.130610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57408 (local address 127.16.43.188:44961)
0620 09:30:36.130812 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:36.130816 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:36.130838 (+ 22us) server_negotiation.cc:407] Connection header received
0620 09:30:36.130864 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:36.130867 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:36.130894 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:36.130983 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:30:36.131719 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:36.132890 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:36.133840 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:36.134266 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:36.134345 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:36.137361 (+ 3016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:30:36.137370 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:30:36.137371 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:30:36.137379 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:30:36.139013 (+ 1634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:30:36.139383 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:30:36.139386 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:30:36.139387 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:30:36.139434 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:30:36.139668 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:30:36.139669 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:30:36.139669 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:30:36.139822 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:30:36.139970 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:36.140063 (+ 93us) server_negotiation.cc:299] Negotiation successful
0620 09:30:36.140127 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":120,"threads_started":1}
I20260620 09:30:36.140350 6910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:36.129306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.190:34499 (local address 127.0.0.1:45606)
0620 09:30:36.129494 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0620 09:30:36.129537 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 09:30:36.130080 (+ 543us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:30:36.130423 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:30:36.130427 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:30:36.130820 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:30:36.131098 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:30:36.131102 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:30:36.132003 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:30:36.132004 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:30:36.133085 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:30:36.133087 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:30:36.133568 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:30:36.133569 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:30:36.133685 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:36.134268 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:30:36.134279 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:30:36.136143 (+ 1864us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:30:36.138549 (+ 2406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:30:36.138580 (+ 31us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:30:36.138582 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:30:36.138896 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:30:36.139168 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:30:36.139168 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:30:36.139169 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:30:36.139274 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:30:36.139594 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:30:36.139599 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:30:36.139769 (+ 170us) client_negotiation.cc:769] Sending connection context
0620 09:30:36.139891 (+ 122us) client_negotiation.cc:240] Negotiation successful
0620 09:30:36.140060 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":102,"threads_started":1}
I20260620 09:30:36.141412 6908 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.43.189" port: 38581 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 09:30:36 dist-test-slave-706q krb5kdc[6677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947836, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.188@KRBTEST.COM for kudu/127.16.43.189@KRBTEST.COM
Jun 20 09:30:36 dist-test-slave-706q krb5kdc[6677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947835, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.43.190@KRBTEST.COM
I20260620 09:30:36.148607 6910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:36.141698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.189:38581 (local address 127.0.0.1:42978)
0620 09:30:36.141741 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0620 09:30:36.141750 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 09:30:36.141780 (+ 30us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:30:36.142000 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:30:36.142001 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:30:36.142142 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:30:36.142235 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:30:36.142237 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:30:36.142892 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:30:36.142893 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:30:36.143370 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:30:36.143372 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:30:36.143867 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:30:36.143868 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:30:36.143931 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:36.144474 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:30:36.144479 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:30:36.145681 (+ 1202us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:30:36.147500 (+ 1819us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:30:36.147501 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:30:36.147502 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:30:36.147749 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:30:36.147975 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:30:36.147976 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:30:36.147977 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:30:36.148054 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:30:36.148315 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:30:36.148316 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:30:36.148370 (+ 54us) client_negotiation.cc:769] Sending connection context
0620 09:30:36.148440 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 09:30:36.148536 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260620 09:30:36.148669 6838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:36.141792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42978 (local address 127.16.43.189:38581)
0620 09:30:36.141868 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:36.141870 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:36.141878 (+ 8us) server_negotiation.cc:407] Connection header received
0620 09:30:36.141920 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:36.141921 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:36.141928 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:36.141997 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:30:36.142323 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:36.142807 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:36.143490 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:36.143784 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:36.143850 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:36.145773 (+ 1923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:30:36.145775 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:30:36.145775 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:30:36.145784 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:30:36.147414 (+ 1630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:30:36.147830 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:30:36.147830 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:30:36.147831 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:30:36.147885 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:30:36.148137 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:30:36.148138 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:30:36.148138 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:30:36.148248 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:30:36.148302 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:36.148484 (+ 182us) server_negotiation.cc:299] Negotiation successful
0620 09:30:36.148561 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 09:30:36.149001 6908 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.43.188" port: 44961 } has no permanent_uuid. Determining permanent_uuid...
I20260620 09:30:36.149515 6761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:36.141495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45610 (local address 127.16.43.190:34499)
0620 09:30:36.141549 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:36.141551 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:36.141559 (+ 8us) server_negotiation.cc:407] Connection header received
0620 09:30:36.141643 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:36.141644 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:36.141656 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:36.141716 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:30:36.142207 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:36.142728 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:36.143443 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:36.143708 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:36.143760 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:36.146354 (+ 2594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:30:36.146355 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:30:36.146356 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:30:36.146364 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:30:36.148327 (+ 1963us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:30:36.148746 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:30:36.148748 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:30:36.148748 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:30:36.148807 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:30:36.149014 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:30:36.149014 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:30:36.149015 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:30:36.149141 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:30:36.149309 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:36.149393 (+ 84us) server_negotiation.cc:299] Negotiation successful
0620 09:30:36.149457 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
Jun 20 09:30:36 dist-test-slave-706q krb5kdc[6677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947836, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.188@KRBTEST.COM for kudu/127.16.43.188@KRBTEST.COM
Jun 20 09:30:36 dist-test-slave-706q krb5kdc[6677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947835, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.43.189@KRBTEST.COM
I20260620 09:30:36.155756 6910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:36.149273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.188:44961 (local address 127.0.0.1:57416)
0620 09:30:36.149391 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0620 09:30:36.149401 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 09:30:36.149442 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:30:36.149594 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:30:36.149595 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:30:36.149722 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:30:36.149751 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:30:36.149752 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:30:36.150595 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:30:36.150596 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:30:36.151072 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:30:36.151073 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:30:36.151454 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:30:36.151455 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:30:36.151504 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:36.151946 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:30:36.151951 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:30:36.153109 (+ 1158us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:30:36.154716 (+ 1607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:30:36.154718 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:30:36.154718 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:30:36.154939 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:30:36.155169 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:30:36.155170 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:30:36.155170 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:30:36.155223 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:30:36.155483 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:30:36.155483 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:30:36.155522 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 09:30:36.155608 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 09:30:36.155688 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110}
I20260620 09:30:36.155864 6912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:36.149331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57416 (local address 127.16.43.188:44961)
0620 09:30:36.149371 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:36.149373 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:36.149490 (+ 117us) server_negotiation.cc:407] Connection header received
0620 09:30:36.149506 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:36.149507 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:36.149518 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:36.149559 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:30:36.149828 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:36.150507 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:36.151153 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:36.151370 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:36.151455 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:36.153191 (+ 1736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:30:36.153192 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:30:36.153192 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:30:36.153200 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:30:36.154632 (+ 1432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:30:36.155022 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:30:36.155023 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:30:36.155023 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:30:36.155090 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:30:36.155304 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:30:36.155305 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:30:36.155305 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:30:36.155415 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:30:36.155492 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:36.155647 (+ 155us) server_negotiation.cc:299] Negotiation successful
0620 09:30:36.155762 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":32,"server-negotiator.queue_time_us":22}
I20260620 09:30:36.156880 6758 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.43.188" port: 44961 } attempt: 2
I20260620 09:30:36.157850 6908 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48ca9acf3b484e8fb17801e2a0126955: Bootstrap starting.
I20260620 09:30:36.158061 6838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:36.150331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42992 (local address 127.16.43.189:38581)
0620 09:30:36.150367 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:36.150369 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:36.150376 (+ 7us) server_negotiation.cc:407] Connection header received
0620 09:30:36.150419 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:36.150420 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:36.150429 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:36.150472 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:30:36.150944 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:36.151395 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:36.152121 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:36.152367 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:36.152434 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:36.155027 (+ 2593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:30:36.155028 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:30:36.155028 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:30:36.155036 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:30:36.156827 (+ 1791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:30:36.157220 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:30:36.157222 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:30:36.157222 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:30:36.157284 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:30:36.157519 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:30:36.157521 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:30:36.157521 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:30:36.157660 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:30:36.157840 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:36.157876 (+ 36us) server_negotiation.cc:299] Negotiation successful
0620 09:30:36.157950 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 09:30:36.158838 6908 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 48ca9acf3b484e8fb17801e2a0126955: Neither blocks nor log segments found. Creating new log.
I20260620 09:30:36.159060 6908 log.cc:824] T 00000000000000000000000000000000 P 48ca9acf3b484e8fb17801e2a0126955: Log is configured to *not* fsync() on all Append() calls
I20260620 09:30:36.160219 6908 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48ca9acf3b484e8fb17801e2a0126955: No bootstrap required, opened a new log
Jun 20 09:30:36 dist-test-slave-706q krb5kdc[6677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947835, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.190@KRBTEST.COM for kudu/127.16.43.188@KRBTEST.COM
I20260620 09:30:36.161434 16558 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 09:30:36.162333 6908 raft_consensus.cc:348] T 00000000000000000000000000000000 P 48ca9acf3b484e8fb17801e2a0126955 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a15b670a7b84fdf832af1cdbc1cf070" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 34499 } } peers { permanent_uuid: "fedac94dd8c8486c939f8f9ee52ba284" member_type: VOTER last_known_addr { host: "127.16.43.189" port: 38581 } } peers { permanent_uuid: "48ca9acf3b484e8fb17801e2a0126955" member_type: VOTER last_known_addr { host: "127.16.43.188" port: 44961 } }
I20260620 09:30:36.162490 6908 raft_consensus.cc:729] T 00000000000000000000000000000000 P 48ca9acf3b484e8fb17801e2a0126955 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48ca9acf3b484e8fb17801e2a0126955, State: Initialized, Role: FOLLOWER
I20260620 09:30:36.162590 6908 consensus_queue.cc:260] T 00000000000000000000000000000000 P 48ca9acf3b484e8fb17801e2a0126955 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a15b670a7b84fdf832af1cdbc1cf070" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 34499 } } peers { permanent_uuid: "fedac94dd8c8486c939f8f9ee52ba284" member_type: VOTER last_known_addr { host: "127.16.43.189" port: 38581 } } peers { permanent_uuid: "48ca9acf3b484e8fb17801e2a0126955" member_type: VOTER last_known_addr { host: "127.16.43.188" port: 44961 } }
I20260620 09:30:36.162909 6908 sys_catalog.cc:564] T 00000000000000000000000000000000 P 48ca9acf3b484e8fb17801e2a0126955 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:30:36.162374 6732 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "1a15b670a7b84fdf832af1cdbc1cf070"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45610
I20260620 09:30:36.163956 6914 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48ca9acf3b484e8fb17801e2a0126955 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a15b670a7b84fdf832af1cdbc1cf070" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 34499 } } peers { permanent_uuid: "fedac94dd8c8486c939f8f9ee52ba284" member_type: VOTER last_known_addr { host: "127.16.43.189" port: 38581 } } peers { permanent_uuid: "48ca9acf3b484e8fb17801e2a0126955" member_type: VOTER last_known_addr { host: "127.16.43.188" port: 44961 } } }
I20260620 09:30:36.164256 6914 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48ca9acf3b484e8fb17801e2a0126955 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 09:30:36.164736 6760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:36.157183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.188:44961 (local address 127.0.0.1:57422)
0620 09:30:36.157239 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0620 09:30:36.157249 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 09:30:36.157285 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:30:36.157576 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:30:36.157577 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:30:36.157735 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:30:36.157785 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:30:36.157788 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:30:36.158678 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:30:36.158679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:30:36.159162 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:30:36.159164 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:30:36.159671 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:30:36.159672 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:30:36.159747 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:36.160167 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:30:36.160172 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:30:36.161691 (+ 1519us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:30:36.163587 (+ 1896us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:30:36.163588 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:30:36.163589 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:30:36.163798 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:30:36.164044 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:30:36.164045 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:30:36.164045 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:30:36.164108 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:30:36.164420 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:30:36.164421 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:30:36.164473 (+ 52us) client_negotiation.cc:769] Sending connection context
0620 09:30:36.164523 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 09:30:36.164629 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
W20260620 09:30:36.164855 6925 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 48ca9acf3b484e8fb17801e2a0126955: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 09:30:36.164898 6925 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 09:30:36.165508 6912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:36.157359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57422 (local address 127.16.43.188:44961)
0620 09:30:36.157427 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:36.157428 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:36.157434 (+ 6us) server_negotiation.cc:407] Connection header received
0620 09:30:36.157449 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:36.157450 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:36.157457 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:36.157566 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:30:36.157873 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:36.158592 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:36.159252 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:36.159584 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:36.159891 (+ 307us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:36.161782 (+ 1891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:30:36.161784 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:30:36.161784 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:30:36.161791 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:30:36.163459 (+ 1668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:30:36.163892 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:30:36.163893 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:30:36.163893 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:30:36.163963 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:30:36.164179 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:30:36.164180 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:30:36.164180 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:30:36.164332 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:30:36.165305 (+ 973us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:36.165347 (+ 42us) server_negotiation.cc:299] Negotiation successful
0620 09:30:36.165425 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 09:30:36.167348 6758 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a15b670a7b84fdf832af1cdbc1cf070: Bootstrap starting.
I20260620 09:30:36.168156 6758 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1a15b670a7b84fdf832af1cdbc1cf070: Neither blocks nor log segments found. Creating new log.
I20260620 09:30:36.168464 6758 log.cc:824] T 00000000000000000000000000000000 P 1a15b670a7b84fdf832af1cdbc1cf070: Log is configured to *not* fsync() on all Append() calls
I20260620 09:30:36.165412 6840 master_runner.cc:186] Error getting master registration for 127.16.43.190:34499: OK, instance_id { permanent_uuid: "1a15b670a7b84fdf832af1cdbc1cf070" instance_seqno: 1781947835811019 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 09:30:36.168659 6840 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 09:30:36.169322 6758 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a15b670a7b84fdf832af1cdbc1cf070: No bootstrap required, opened a new log
I20260620 09:30:36.170655 6758 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1a15b670a7b84fdf832af1cdbc1cf070 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a15b670a7b84fdf832af1cdbc1cf070" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 34499 } } peers { permanent_uuid: "fedac94dd8c8486c939f8f9ee52ba284" member_type: VOTER last_known_addr { host: "127.16.43.189" port: 38581 } } peers { permanent_uuid: "48ca9acf3b484e8fb17801e2a0126955" member_type: VOTER last_known_addr { host: "127.16.43.188" port: 44961 } }
I20260620 09:30:36.170792 6758 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1a15b670a7b84fdf832af1cdbc1cf070 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a15b670a7b84fdf832af1cdbc1cf070, State: Initialized, Role: FOLLOWER
I20260620 09:30:36.170862 6758 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a15b670a7b84fdf832af1cdbc1cf070 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a15b670a7b84fdf832af1cdbc1cf070" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 34499 } } peers { permanent_uuid: "fedac94dd8c8486c939f8f9ee52ba284" member_type: VOTER last_known_addr { host: "127.16.43.189" port: 38581 } } peers { permanent_uuid: "48ca9acf3b484e8fb17801e2a0126955" member_type: VOTER last_known_addr { host: "127.16.43.188" port: 44961 } }
I20260620 09:30:36.171183 6758 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1a15b670a7b84fdf832af1cdbc1cf070 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:30:36.171137 6928 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a15b670a7b84fdf832af1cdbc1cf070 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a15b670a7b84fdf832af1cdbc1cf070" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 34499 } } peers { permanent_uuid: "fedac94dd8c8486c939f8f9ee52ba284" member_type: VOTER last_known_addr { host: "127.16.43.189" port: 38581 } } peers { permanent_uuid: "48ca9acf3b484e8fb17801e2a0126955" member_type: VOTER last_known_addr { host: "127.16.43.188" port: 44961 } } }
I20260620 09:30:36.171249 6928 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a15b670a7b84fdf832af1cdbc1cf070 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 09:30:36.171311 6732 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1a15b670a7b84fdf832af1cdbc1cf070 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 09:30:36.171375 6732 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1a15b670a7b84fdf832af1cdbc1cf070 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:30:36.172303 6732 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1a15b670a7b84fdf832af1cdbc1cf070 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a15b670a7b84fdf832af1cdbc1cf070" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 34499 } } peers { permanent_uuid: "fedac94dd8c8486c939f8f9ee52ba284" member_type: VOTER last_known_addr { host: "127.16.43.189" port: 38581 } } peers { permanent_uuid: "48ca9acf3b484e8fb17801e2a0126955" member_type: VOTER last_known_addr { host: "127.16.43.188" port: 44961 } }
I20260620 09:30:36.172590 6732 leader_election.cc:288] T 00000000000000000000000000000000 P 1a15b670a7b84fdf832af1cdbc1cf070 [CANDIDATE]: Term 1 election: Requested vote from peers fedac94dd8c8486c939f8f9ee52ba284 (127.16.43.189:38581), 48ca9acf3b484e8fb17801e2a0126955 (127.16.43.188:44961)
I20260620 09:30:36.173786 6882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1a15b670a7b84fdf832af1cdbc1cf070" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "48ca9acf3b484e8fb17801e2a0126955"
I20260620 09:30:36.173919 6882 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 48ca9acf3b484e8fb17801e2a0126955 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:30:36.175239 6882 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 48ca9acf3b484e8fb17801e2a0126955 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a15b670a7b84fdf832af1cdbc1cf070 in term 1.
I20260620 09:30:36.175935 6706 leader_election.cc:302] T 00000000000000000000000000000000 P 1a15b670a7b84fdf832af1cdbc1cf070 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a15b670a7b84fdf832af1cdbc1cf070, 48ca9acf3b484e8fb17801e2a0126955; no voters:
I20260620 09:30:36.176054 6928 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1a15b670a7b84fdf832af1cdbc1cf070 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:30:36.176187 6928 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1a15b670a7b84fdf832af1cdbc1cf070 [term 1 LEADER]: Becoming Leader. State: Replica: 1a15b670a7b84fdf832af1cdbc1cf070, State: Running, Role: LEADER
I20260620 09:30:36.176263 6928 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a15b670a7b84fdf832af1cdbc1cf070 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a15b670a7b84fdf832af1cdbc1cf070" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 34499 } } peers { permanent_uuid: "fedac94dd8c8486c939f8f9ee52ba284" member_type: VOTER last_known_addr { host: "127.16.43.189" port: 38581 } } peers { permanent_uuid: "48ca9acf3b484e8fb17801e2a0126955" member_type: VOTER last_known_addr { host: "127.16.43.188" port: 44961 } }
I20260620 09:30:36.176649 6928 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a15b670a7b84fdf832af1cdbc1cf070 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a15b670a7b84fdf832af1cdbc1cf070. Latest consensus state: current_term: 1 leader_uuid: "1a15b670a7b84fdf832af1cdbc1cf070" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a15b670a7b84fdf832af1cdbc1cf070" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 34499 } } peers { permanent_uuid: "fedac94dd8c8486c939f8f9ee52ba284" member_type: VOTER last_known_addr { host: "127.16.43.189" port: 38581 } } peers { permanent_uuid: "48ca9acf3b484e8fb17801e2a0126955" member_type: VOTER last_known_addr { host: "127.16.43.188" port: 44961 } } }
I20260620 09:30:36.176726 6928 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a15b670a7b84fdf832af1cdbc1cf070 [sys.catalog]: This master's current role is: LEADER
I20260620 09:30:36.177086 6949 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:30:36.177593 6949 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:30:36.175832 6690 master_runner.cc:186] Error getting master registration for 127.16.43.189:38581: OK, instance_id { permanent_uuid: "fedac94dd8c8486c939f8f9ee52ba284" instance_seqno: 1781947835964646 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 09:30:36.179023 6690 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 09:30:36.182849 6838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:36.174151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43004 (local address 127.16.43.189:38581)
0620 09:30:36.174197 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:36.174199 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:36.174208 (+ 9us) server_negotiation.cc:407] Connection header received
0620 09:30:36.174233 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:36.174234 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:36.174244 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:36.174294 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:30:36.175137 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:36.175965 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:36.177048 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:36.177326 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:36.177371 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:36.179149 (+ 1778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:30:36.179151 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:30:36.179151 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:30:36.179159 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:30:36.181542 (+ 2383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:30:36.181941 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:30:36.181942 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:30:36.181943 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:30:36.182000 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:30:36.182198 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:30:36.182199 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:30:36.182200 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:30:36.182325 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:30:36.182399 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:36.182618 (+ 219us) server_negotiation.cc:299] Negotiation successful
0620 09:30:36.182734 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 09:30:36.187333 6882 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 48ca9acf3b484e8fb17801e2a0126955 [term 1 FOLLOWER]: Refusing update from remote peer 1a15b670a7b84fdf832af1cdbc1cf070: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:30:36.188503 6947 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1a15b670a7b84fdf832af1cdbc1cf070 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48ca9acf3b484e8fb17801e2a0126955" member_type: VOTER last_known_addr { host: "127.16.43.188" port: 44961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:30:36.190088 6912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:36.174232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57426 (local address 127.16.43.188:44961)
0620 09:30:36.175585 (+ 1353us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:36.175587 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:36.175596 (+ 9us) server_negotiation.cc:407] Connection header received
0620 09:30:36.175617 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:36.175617 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:36.175627 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:36.175686 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:30:36.179459 (+ 3773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:36.180213 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:36.183841 (+ 3628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:36.184125 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:36.184167 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:36.186451 (+ 2284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:30:36.186452 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:30:36.186452 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:30:36.186460 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:30:36.188332 (+ 1872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:30:36.188947 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:30:36.188949 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:30:36.188949 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:30:36.189023 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:30:36.189326 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:30:36.189327 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:30:36.189327 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:30:36.189470 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:30:36.189540 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:36.189907 (+ 367us) server_negotiation.cc:299] Negotiation successful
0620 09:30:36.189987 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1333}
I20260620 09:30:36.190404 6914 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48ca9acf3b484e8fb17801e2a0126955 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a15b670a7b84fdf832af1cdbc1cf070. Latest consensus state: current_term: 1 leader_uuid: "1a15b670a7b84fdf832af1cdbc1cf070" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a15b670a7b84fdf832af1cdbc1cf070" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 34499 } } peers { permanent_uuid: "fedac94dd8c8486c939f8f9ee52ba284" member_type: VOTER last_known_addr { host: "127.16.43.189" port: 38581 } } peers { permanent_uuid: "48ca9acf3b484e8fb17801e2a0126955" member_type: VOTER last_known_addr { host: "127.16.43.188" port: 44961 } } }
I20260620 09:30:36.190541 6914 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48ca9acf3b484e8fb17801e2a0126955 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 09:30:36.191910 6761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:36.174420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45620 (local address 127.16.43.190:34499)
0620 09:30:36.174456 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:36.174458 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:36.174827 (+ 369us) server_negotiation.cc:407] Connection header received
0620 09:30:36.175120 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:36.175121 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:36.175133 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:36.175177 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:30:36.180472 (+ 5295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:36.181212 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:36.182819 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:36.183112 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:36.183157 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:36.185976 (+ 2819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:30:36.185977 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:30:36.185978 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:30:36.185986 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:30:36.188401 (+ 2415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:30:36.190914 (+ 2513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:30:36.190915 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:30:36.190915 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:30:36.190975 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:30:36.191198 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:30:36.191199 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:30:36.191200 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:30:36.191343 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:30:36.191437 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:36.191718 (+ 281us) server_negotiation.cc:299] Negotiation successful
0620 09:30:36.191798 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 09:30:36.193338 6928 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a15b670a7b84fdf832af1cdbc1cf070 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1a15b670a7b84fdf832af1cdbc1cf070" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a15b670a7b84fdf832af1cdbc1cf070" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 34499 } } peers { permanent_uuid: "fedac94dd8c8486c939f8f9ee52ba284" member_type: VOTER last_known_addr { host: "127.16.43.189" port: 38581 } } peers { permanent_uuid: "48ca9acf3b484e8fb17801e2a0126955" member_type: VOTER last_known_addr { host: "127.16.43.188" port: 44961 } } }
I20260620 09:30:36.193434 6928 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a15b670a7b84fdf832af1cdbc1cf070 [sys.catalog]: This master's current role is: LEADER
I20260620 09:30:36.193506 6914 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48ca9acf3b484e8fb17801e2a0126955 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1a15b670a7b84fdf832af1cdbc1cf070" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a15b670a7b84fdf832af1cdbc1cf070" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 34499 } } peers { permanent_uuid: "fedac94dd8c8486c939f8f9ee52ba284" member_type: VOTER last_known_addr { host: "127.16.43.189" port: 38581 } } peers { permanent_uuid: "48ca9acf3b484e8fb17801e2a0126955" member_type: VOTER last_known_addr { host: "127.16.43.188" port: 44961 } } }
I20260620 09:30:36.193588 6914 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48ca9acf3b484e8fb17801e2a0126955 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 09:30:36.194290 6951 mvcc.cc:205] Tried to move back new op lower bound from 7298858337018974208 to 7298858336978685952. Current Snapshot: MvccSnapshot[applied={T|T < 7298858337018974208 or (T in {7298858337018974208})}]
I20260620 09:30:36.194476 6949 catalog_manager.cc:1347] Generated new cluster ID: 254a94c99c53443183a30a6e86a7f68f
I20260620 09:30:36.194515 6949 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:30:36.200246 6952 mvcc.cc:205] Tried to move back new op lower bound from 7298858337018974208 to 7298858336978685952. Current Snapshot: MvccSnapshot[applied={T|T < 7298858337018974208}]
I20260620 09:30:36.201649 6833 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.43.188" port: 44961 } attempt: 3
Jun 20 09:30:36 dist-test-slave-706q krb5kdc[6677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947835, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.43.189@KRBTEST.COM for kudu/127.16.43.188@KRBTEST.COM
I20260620 09:30:36.210009 6836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:36.201943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.43.188:44961 (local address 127.0.0.1:57432)
0620 09:30:36.201994 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0620 09:30:36.202004 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 09:30:36.202045 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:30:36.202223 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:30:36.202224 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:30:36.202413 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:30:36.202464 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:30:36.202466 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:30:36.203330 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:30:36.203331 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:30:36.204184 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:30:36.204187 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:30:36.204727 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:30:36.204728 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:30:36.204792 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:36.205257 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:30:36.205262 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:30:36.207109 (+ 1847us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:30:36.208911 (+ 1802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:30:36.208913 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:30:36.208913 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:30:36.209122 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:30:36.209341 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:30:36.209342 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:30:36.209342 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:30:36.209407 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:30:36.209698 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:30:36.209699 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:30:36.209755 (+ 56us) client_negotiation.cc:769] Sending connection context
0620 09:30:36.209811 (+ 56us) client_negotiation.cc:240] Negotiation successful
0620 09:30:36.209907 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260620 09:30:36.210398 6912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:36.201994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57432 (local address 127.16.43.188:44961)
0620 09:30:36.202056 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:36.202057 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:36.202066 (+ 9us) server_negotiation.cc:407] Connection header received
0620 09:30:36.202124 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:36.202125 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:36.202135 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:36.202182 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:30:36.202524 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:36.203229 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:36.204301 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:36.204608 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:36.204660 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:36.207205 (+ 2545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:30:36.207206 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:30:36.207207 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:30:36.207214 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:30:36.208801 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:30:36.209202 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:30:36.209203 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:30:36.209203 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:30:36.209267 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:30:36.209471 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:30:36.209472 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:30:36.209473 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:30:36.209613 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:30:36.209662 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:36.210233 (+ 571us) server_negotiation.cc:299] Negotiation successful
0620 09:30:36.210309 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 09:30:36.211951 6833 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fedac94dd8c8486c939f8f9ee52ba284: Bootstrap starting.
I20260620 09:30:36.212684 6833 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fedac94dd8c8486c939f8f9ee52ba284: Neither blocks nor log segments found. Creating new log.
I20260620 09:30:36.212972 6833 log.cc:824] T 00000000000000000000000000000000 P fedac94dd8c8486c939f8f9ee52ba284: Log is configured to *not* fsync() on all Append() calls
I20260620 09:30:36.228391 6833 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fedac94dd8c8486c939f8f9ee52ba284: No bootstrap required, opened a new log
I20260620 09:30:36.231498 6833 raft_consensus.cc:348] T 00000000000000000000000000000000 P fedac94dd8c8486c939f8f9ee52ba284 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a15b670a7b84fdf832af1cdbc1cf070" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 34499 } } peers { permanent_uuid: "fedac94dd8c8486c939f8f9ee52ba284" member_type: VOTER last_known_addr { host: "127.16.43.189" port: 38581 } } peers { permanent_uuid: "48ca9acf3b484e8fb17801e2a0126955" member_type: VOTER last_known_addr { host: "127.16.43.188" port: 44961 } }
I20260620 09:30:36.231890 6833 raft_consensus.cc:729] T 00000000000000000000000000000000 P fedac94dd8c8486c939f8f9ee52ba284 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fedac94dd8c8486c939f8f9ee52ba284, State: Initialized, Role: FOLLOWER
I20260620 09:30:36.232033 6833 consensus_queue.cc:260] T 00000000000000000000000000000000 P fedac94dd8c8486c939f8f9ee52ba284 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a15b670a7b84fdf832af1cdbc1cf070" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 34499 } } peers { permanent_uuid: "fedac94dd8c8486c939f8f9ee52ba284" member_type: VOTER last_known_addr { host: "127.16.43.189" port: 38581 } } peers { permanent_uuid: "48ca9acf3b484e8fb17801e2a0126955" member_type: VOTER last_known_addr { host: "127.16.43.188" port: 44961 } }
I20260620 09:30:36.232378 6957 sys_catalog.cc:455] T 00000000000000000000000000000000 P fedac94dd8c8486c939f8f9ee52ba284 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a15b670a7b84fdf832af1cdbc1cf070" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 34499 } } peers { permanent_uuid: "fedac94dd8c8486c939f8f9ee52ba284" member_type: VOTER last_known_addr { host: "127.16.43.189" port: 38581 } } peers { permanent_uuid: "48ca9acf3b484e8fb17801e2a0126955" member_type: VOTER last_known_addr { host: "127.16.43.188" port: 44961 } } }
I20260620 09:30:36.232663 6957 sys_catalog.cc:458] T 00000000000000000000000000000000 P fedac94dd8c8486c939f8f9ee52ba284 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 09:30:36.232517 6807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1a15b670a7b84fdf832af1cdbc1cf070" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "fedac94dd8c8486c939f8f9ee52ba284"
I20260620 09:30:36.232492 6806 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fedac94dd8c8486c939f8f9ee52ba284 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:30:36.232431 6833 sys_catalog.cc:564] T 00000000000000000000000000000000 P fedac94dd8c8486c939f8f9ee52ba284 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:30:36.233606 6806 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fedac94dd8c8486c939f8f9ee52ba284 [term 1 FOLLOWER]: Refusing update from remote peer 1a15b670a7b84fdf832af1cdbc1cf070: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:30:36.233974 6928 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1a15b670a7b84fdf832af1cdbc1cf070 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fedac94dd8c8486c939f8f9ee52ba284" member_type: VOTER last_known_addr { host: "127.16.43.189" port: 38581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:30:36.235177 6949 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:30:36.235244 6807 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fedac94dd8c8486c939f8f9ee52ba284 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a15b670a7b84fdf832af1cdbc1cf070 in term 1.
I20260620 09:30:36.236001 6949 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:30:36.237154 6957 sys_catalog.cc:455] T 00000000000000000000000000000000 P fedac94dd8c8486c939f8f9ee52ba284 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a15b670a7b84fdf832af1cdbc1cf070. Latest consensus state: current_term: 1 leader_uuid: "1a15b670a7b84fdf832af1cdbc1cf070" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a15b670a7b84fdf832af1cdbc1cf070" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 34499 } } peers { permanent_uuid: "fedac94dd8c8486c939f8f9ee52ba284" member_type: VOTER last_known_addr { host: "127.16.43.189" port: 38581 } } peers { permanent_uuid: "48ca9acf3b484e8fb17801e2a0126955" member_type: VOTER last_known_addr { host: "127.16.43.188" port: 44961 } } }
I20260620 09:30:36.240406 6949 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1a15b670a7b84fdf832af1cdbc1cf070: Generated new TSK 0
I20260620 09:30:36.240602 6949 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 09:30:36.241031 6957 sys_catalog.cc:458] T 00000000000000000000000000000000 P fedac94dd8c8486c939f8f9ee52ba284 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 09:30:36.241171 6957 sys_catalog.cc:455] T 00000000000000000000000000000000 P fedac94dd8c8486c939f8f9ee52ba284 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1a15b670a7b84fdf832af1cdbc1cf070" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a15b670a7b84fdf832af1cdbc1cf070" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 34499 } } peers { permanent_uuid: "fedac94dd8c8486c939f8f9ee52ba284" member_type: VOTER last_known_addr { host: "127.16.43.189" port: 38581 } } peers { permanent_uuid: "48ca9acf3b484e8fb17801e2a0126955" member_type: VOTER last_known_addr { host: "127.16.43.188" port: 44961 } } }
I20260620 09:30:36.241245 6957 sys_catalog.cc:458] T 00000000000000000000000000000000 P fedac94dd8c8486c939f8f9ee52ba284 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 09:30:36.244081 6968 mvcc.cc:205] Tried to move back new op lower bound from 7298858337199280128 to 7298858336978685952. Current Snapshot: MvccSnapshot[applied={T|T < 7298858337018974208}]
W20260620 09:30:36.245164 6971 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fedac94dd8c8486c939f8f9ee52ba284: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 09:30:36.245218 6971 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 09:30:36.247617 6928 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a15b670a7b84fdf832af1cdbc1cf070 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1a15b670a7b84fdf832af1cdbc1cf070" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a15b670a7b84fdf832af1cdbc1cf070" member_type: VOTER last_known_addr { host: "127.16.43.190" port: 34499 } } peers { permanent_uuid: "fedac94dd8c8486c939f8f9ee52ba284" member_type: VOTER last_known_addr { host: "127.16.43.189" port: 38581 } } peers { permanent_uuid: "48ca9acf3b484e8fb17801e2a0126955" member_type: VOTER last_known_addr { host: "127.16.43.188" port: 44961 } } }
I20260620 09:30:36.247742 6928 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a15b670a7b84fdf832af1cdbc1cf070 [sys.catalog]: This master's current role is: LEADER
W20260620 09:30:36.263895 6761 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:30:36.262744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45636 (local address 127.16.43.190:34499)
0620 09:30:36.262790 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:36.262792 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:36.263165 (+ 373us) server_negotiation.cc:407] Connection header received
0620 09:30:36.263281 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:36.263282 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:36.263295 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:36.263328 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:30:36.263819 (+ 491us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45636: BlockingRecv error: recv got EOF from 127.0.0.1:45636 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260620 09:30:36.265364 6838 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:30:36.264427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43014 (local address 127.16.43.189:38581)
0620 09:30:36.264468 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:36.264471 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:36.264774 (+ 303us) server_negotiation.cc:407] Connection header received
0620 09:30:36.264873 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:36.264874 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:36.264885 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:36.264915 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:30:36.265296 (+ 381us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43014: BlockingRecv error: recv got EOF from 127.0.0.1:43014 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260620 09:30:36.266286 6912 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:30:36.264658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57448 (local address 127.16.43.188:44961)
0620 09:30:36.264696 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:36.264698 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:36.265774 (+ 1076us) server_negotiation.cc:407] Connection header received
0620 09:30:36.265869 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:36.265870 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:36.265881 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:36.265910 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:30:36.266226 (+ 316us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57448: BlockingRecv error: recv got EOF from 127.0.0.1:57448 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260620 09:30:36.268831 6838 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:30:36.267958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43026 (local address 127.16.43.189:38581)
0620 09:30:36.268002 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:36.268005 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:36.268238 (+ 233us) server_negotiation.cc:407] Connection header received
0620 09:30:36.268364 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:36.268366 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:36.268388 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:36.268420 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:30:36.268782 (+ 362us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43026: BlockingRecv error: recv got EOF from 127.0.0.1:43026 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260620 09:30:36.272094 6838 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:30:36.271257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43032 (local address 127.16.43.189:38581)
0620 09:30:36.271291 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:36.271292 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:36.271580 (+ 288us) server_negotiation.cc:407] Connection header received
0620 09:30:36.271681 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:36.271682 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:36.271692 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:36.271720 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:30:36.272044 (+ 324us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43032: BlockingRecv error: recv got EOF from 127.0.0.1:43032 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260620 09:30:36.276197 6838 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:30:36.275415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43034 (local address 127.16.43.189:38581)
0620 09:30:36.275451 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:36.275452 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:36.275691 (+ 239us) server_negotiation.cc:407] Connection header received
0620 09:30:36.275786 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:36.275787 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:36.275797 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:36.275825 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:30:36.276146 (+ 321us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43034: BlockingRecv error: recv got EOF from 127.0.0.1:43034 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260620 09:30:36.282469 6838 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:30:36.281614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43046 (local address 127.16.43.189:38581)
0620 09:30:36.281652 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:36.281654 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:36.281868 (+ 214us) server_negotiation.cc:407] Connection header received
0620 09:30:36.281986 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:36.281987 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:36.281998 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:36.282027 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:30:36.282380 (+ 353us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43046: BlockingRecv error: recv got EOF from 127.0.0.1:43046 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260620 09:30:36.292881 6838 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:30:36.291944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43062 (local address 127.16.43.189:38581)
0620 09:30:36.291987 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:36.291989 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:36.292309 (+ 320us) server_negotiation.cc:407] Connection header received
0620 09:30:36.292411 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:36.292413 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:36.292426 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:36.292458 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:30:36.292819 (+ 361us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43062: BlockingRecv error: recv got EOF from 127.0.0.1:43062 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260620 09:30:36.311414 6838 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:30:36.310448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50626 (local address 127.16.43.189:38581)
0620 09:30:36.310495 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:36.310497 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:36.310779 (+ 282us) server_negotiation.cc:407] Connection header received
0620 09:30:36.310896 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:36.310898 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:36.310911 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:36.310943 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:30:36.311354 (+ 411us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50626: BlockingRecv error: recv got EOF from 127.0.0.1:50626 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260620 09:30:36.346135 6838 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:30:36.345119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50628 (local address 127.16.43.189:38581)
0620 09:30:36.345220 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:36.345222 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:36.345498 (+ 276us) server_negotiation.cc:407] Connection header received
0620 09:30:36.345608 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:36.345610 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:36.345622 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:36.345653 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:30:36.346072 (+ 419us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50628: BlockingRecv error: recv got EOF from 127.0.0.1:50628 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260620 09:30:36.412439 6838 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:30:36.411767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50638 (local address 127.16.43.189:38581)
0620 09:30:36.411841 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:36.411843 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:36.411986 (+ 143us) server_negotiation.cc:407] Connection header received
0620 09:30:36.412013 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:36.412014 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:36.412025 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:36.412067 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:30:36.412389 (+ 322us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50638: BlockingRecv error: recv got EOF from 127.0.0.1:50638 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260620 09:30:36.542779 6838 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:30:36.541949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50646 (local address 127.16.43.189:38581)
0620 09:30:36.542044 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:36.542046 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:36.542175 (+ 129us) server_negotiation.cc:407] Connection header received
0620 09:30:36.542273 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:36.542275 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:36.542294 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:36.542365 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:30:36.542732 (+ 367us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50646: BlockingRecv error: recv got EOF from 127.0.0.1:50646 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260620 09:30:36.800856 6838 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:30:36.800309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50648 (local address 127.16.43.189:38581)
0620 09:30:36.800385 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:36.800387 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:36.800396 (+ 9us) server_negotiation.cc:407] Connection header received
0620 09:30:36.800422 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:36.800423 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:36.800434 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:36.800484 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:30:36.800805 (+ 321us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50648: BlockingRecv error: recv got EOF from 127.0.0.1:50648 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 09:30:37.165362 6925 catalog_manager.cc:1259] Loaded cluster ID: 254a94c99c53443183a30a6e86a7f68f
I20260620 09:30:37.165442 6925 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 48ca9acf3b484e8fb17801e2a0126955: loading cluster ID for follower catalog manager: success
I20260620 09:30:37.166250 6925 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 48ca9acf3b484e8fb17801e2a0126955: acquiring CA information for follower catalog manager: success
I20260620 09:30:37.166504 6925 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 48ca9acf3b484e8fb17801e2a0126955: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 09:30:37.245734 6971 catalog_manager.cc:1259] Loaded cluster ID: 254a94c99c53443183a30a6e86a7f68f
I20260620 09:30:37.245831 6971 catalog_manager.cc:1552] T 00000000000000000000000000000000 P fedac94dd8c8486c939f8f9ee52ba284: loading cluster ID for follower catalog manager: success
I20260620 09:30:37.246850 6971 catalog_manager.cc:1574] T 00000000000000000000000000000000 P fedac94dd8c8486c939f8f9ee52ba284: acquiring CA information for follower catalog manager: success
I20260620 09:30:37.247265 6971 catalog_manager.cc:1602] T 00000000000000000000000000000000 P fedac94dd8c8486c939f8f9ee52ba284: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 09:30:37.317090 7038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:37.314123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50660 (local address 127.16.43.189:38581)
0620 09:30:37.314220 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:37.314222 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:37.314232 (+ 10us) server_negotiation.cc:407] Connection header received
0620 09:30:37.314283 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:37.314284 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:37.314336 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:37.314420 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:30:37.314736 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:37.315285 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:37.316163 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:37.316388 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:37.316450 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:37.316647 (+ 197us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:30:37.316758 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:30:37.316833 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:37.316976 (+ 143us) server_negotiation.cc:299] Negotiation successful
0620 09:30:37.316998 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81,"thread_start_us":51,"threads_started":1}
I20260620 09:30:37.321799 7044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:37.319216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43990 (local address 127.16.43.188:44961)
0620 09:30:37.319385 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:37.319387 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:37.319416 (+ 29us) server_negotiation.cc:407] Connection header received
0620 09:30:37.319512 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:37.319512 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:37.319620 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:37.319705 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:30:37.319839 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:37.320316 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:37.321069 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:37.321264 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:37.321317 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:37.321464 (+ 147us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:30:37.321564 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:30:37.321619 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:37.321668 (+ 49us) server_negotiation.cc:299] Negotiation successful
0620 09:30:37.321684 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":85,"threads_started":1}
I20260620 09:30:37.326869 7038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:37.323586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50674 (local address 127.16.43.189:38581)
0620 09:30:37.323630 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:37.323632 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:37.324125 (+ 493us) server_negotiation.cc:407] Connection header received
0620 09:30:37.324158 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:37.324159 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:37.324170 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:37.324233 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:30:37.324391 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:37.325096 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:37.326058 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:37.326351 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:37.326410 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:37.326615 (+ 205us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:30:37.326647 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:30:37.326695 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:37.326765 (+ 70us) server_negotiation.cc:299] Negotiation successful
0620 09:30:37.326786 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 09:30:37.326907 7044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:37.323834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44004 (local address 127.16.43.188:44961)
0620 09:30:37.323889 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:37.323890 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:37.324250 (+ 360us) server_negotiation.cc:407] Connection header received
0620 09:30:37.324346 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:37.324347 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:37.324357 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:37.324406 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:30:37.324608 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:37.325283 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:37.326130 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:37.326418 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:37.326478 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:37.326639 (+ 161us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:30:37.326672 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:30:37.326701 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:37.326811 (+ 110us) server_negotiation.cc:299] Negotiation successful
0620 09:30:37.326834 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 09:30:37.326960 7051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:30:37.323650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49332 (local address 127.16.43.190:34499)
0620 09:30:37.323786 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 09:30:37.323788 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:30:37.323795 (+ 7us) server_negotiation.cc:407] Connection header received
0620 09:30:37.323837 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:30:37.323837 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:30:37.323887 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:30:37.323982 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:30:37.324121 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:37.324830 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:37.325851 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:30:37.326173 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:30:37.326256 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:30:37.326437 (+ 181us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:30:37.326563 (+ 126us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:30:37.326621 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 09:30:37.326846 (+ 225us) server_negotiation.cc:299] Negotiation successful
0620 09:30:37.326872 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":48,"threads_started":1}
I20260620 09:30:37.332648 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 6690
I20260620 09:30:37.338280 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 6765
I20260620 09:30:37.342983 16558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMkdNwY/build/release/bin/kudu with pid 6840
2026-06-20T09:30:37Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2832 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5679 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (274401 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST